Compoze

Staff could ask a question and get an answer with its sources attached, so they could open the document and check it instead of searching through files by hand.

Working alone

Each engagement ran from the first customer conversation through deployment and training, and I handled all of it. Organising the knowledge base took the longest, because only the customer could decide which documents belonged in it and which should stay out. Running Compoze beside a full-time job also meant it had to keep working without constant attention.

One codebase for every customer

Compoze ran on two stacks: a Python assistant on LlamaIndex and FastAPI with LlamaCloud doing the indexing, and the platform on this page, built on Next.js and Postgres with pgvector. Every customer shared one codebase and one database. Each customer's rows carry a tenant id and are kept apart twice, once in the app and once with Postgres row-level security. One database kept migrations and billing simple, but it meant the tenant checks had to be right everywhere. Nine models from OpenAI, Anthropic and Google sit behind one gateway, chosen per environment in config.

Finding the right passages

Documents are split into chunks of about a thousand characters, and each chunk gets a one-line header saying where in the document it came from. A question searches up to five knowledge bases at once, using both embeddings and plain keyword search, since embeddings alone miss exact words like codes and names. Anything under a 0.35 similarity score is dropped, and a reranker puts the rest in order.

Each citation is checked against the sentence it supports, and the source links and scores let readers see how well a passage backs up the answer. When even the best passage is under the cutoff, the assistant says it can't answer instead of guessing.

How I knew it worked

I kept a fixed set of questions and expected answers from logs and testing, including questions the documents couldn't answer, where the right reply was a refusal. Search and the written answer are scored separately. The set runs in CI, so a change to the prompt, the chunking or the model fails if it does worse than the version on main.

Getting the documents in

Files come in from Lark and Google Drive, and processing runs in the background on QStash. The failure cases took more of my time than the normal path, because a job that dies halfway leaves a document that looks uploaded but never shows up in answers. So each step can be retried on its own, uploading the same file twice does nothing, and anything that keeps failing lands in a dead-letter queue with the error attached. That gave me a record of what failed and what needed retrying.

Safety

A synced document could contain instructions aimed at the model. I mark retrieved text as data in the prompt and limit each agent to its own tools. That limits what a planted instruction can do, but it doesn't stop the model from reading it.