Compoze

2025 · sole founder · sold to a client

Compoze was an evenings-and-weekends company, run alongside my full-time job at TikTok. It built assistants that answer questions from a company's own documents, and it was sold to a client at the end of 2025.

The idea was to help staff find answers in company documents without searching through them by hand. Compoze included the sources with each answer so people could open them and check.

An answered question in Compoze: a knowledge base search over three documents, a structured answer containing a table, and cited sources listed with match scores. — fig. 1 — the tool call is shown, the answer is structured, and the sources sit under it with their match scores.

Figure 02 · User flow

An admin fills the knowledge base, then a user asks questions

An admin connects Lark or Drive and watches each document until it is ready or fails. A user picks an agent, starts from example prompts, and gets a cited answer, or a refusal when nothing clears the 0.35 floor.

Admin
User
Answer

fig. 2 — two people use it. An admin fills the knowledge base and watches ingest; a user asks and checks the sources. The dashed hop marks the handoff from the admin's side to the user's.

Working alone

Each engagement ran from the first customer conversation through deployment and training, and I handled all of it. Organising the knowledge base took the longest, because only the customer could decide which documents belonged in it and which should stay out. Running Compoze beside a full-time job also meant it had to keep working without constant attention.

Figure 03 · The whole system

How a question and a document move through the system

A question walks agent, retrieval, pgvector, gateway and cited answer inside one request. Ingest runs on QStash workers and lands in the same Postgres.

Request
Response
Ingest

fig. 3 — the whole system. A question walks ① agent → ② retrieval → ③ pgvector → ④ gateway → ⑤ cited answer inside one request. Ingest runs elsewhere: Ⓐ download, Ⓑ extract, Ⓒ chunk, Ⓓ embed, Ⓔ store. Everything sits in one database, with every row under a tenant id and row-level security.

Compoze chat in its empty state, showing the selected agent, four example prompts and the list of tools it can use. — fig. 4 — the empty state. Each agent arrives with its own example prompts and a visible list of what it can reach.

The chat opens with example prompts and a list of the tools it can use, so someone new can see what to ask. Past conversations stay in the sidebar.

One codebase, configured per customer

Compoze ran on two stacks: a Python assistant on LlamaIndex and FastAPI with managed indexing on LlamaCloud, and the platform described here. That platform is Next.js 15 and React 19 on the front, Postgres with pgvector behind Drizzle. Every core table carries a tenant id. Tenant checks in the application and Postgres row-level security provide two layers of isolation. One shared database keeps migrations and billing in one place, while tenant scoping has to stay consistent across both layers. On top: three roles (Admin, Manager, User), an admin portal per tenant, and nine models from OpenAI, Anthropic and Google behind one gateway, picked per environment in configuration.

Finding the right passages

Documents are chunked at about a thousand characters. Each chunk gets a one-sentence header placing it in its document and is embedded with OpenAI at 1536 dimensions. A question runs across up to five knowledge bases in parallel. Each lane is hybrid: cosine search over an HNSW index, keyword search beside it because dense search misses exact tokens, the two ranked lists merged with reciprocal rank fusion. Chunks under a 0.35 similarity floor are dropped. An LLM scores the survivors per database, and a cross-encoder reading question and chunk together sets the final order.

Figure 05 · Query

How a question searches five knowledge bases and checks its citations

A question fans out across five knowledge bases with hybrid search. Only chunks over the 0.35 floor survive, a cross-encoder orders them, and each citation is checked against its chunk before the answer ships.

Question
Retrieval
Answer

fig. 5 — one question, five knowledge bases at once. Only chunks over the 0.35 floor survive, and a cross-encoder orders them.

Citations are stored with the message, checked for entailment against the sentence they support, and rendered with their match scores. The source links and scores let readers check how well a passage supports the answer.

The prompt is fixed blocks (role, rules, context tagged with chunk ids, history, question) so the static prefix caches per tenant. The rules: answer only from the context, name what is missing, and refuse when the best chunk sits under the floor.

How I knew it worked

A fixed set of question and answer pairs came from logs and testing, including questions the documents couldn't answer, where the expected reply was a refusal. Retrieval and generation are scored apart: context precision for whether the good chunks ranked high, faithfulness for whether each claim is entailed by what came back. The set runs in CI, and a prompt, chunker or model change fails on a regression against main. The comparison is against the previous version rather than a fixed score threshold.

Getting the documents in

Files come from Lark and Google Drive over OAuth, with an MCP server for Lark docs, wiki, sheets and bitable. Ingest is async through QStash: download, extract with LlamaParse, chunk, batch-embed, store. The failure paths took more of my time than the happy one, because a run that dies after the download leaves a document that looks uploaded and answers nothing.

Figure 06 · Ingest

How an uploaded document is processed, and where failures go

A content hash drops files already seen. PII comes out before chunking. The dashed region runs later on QStash, and every stage can drop out to a dead-letter queue the admin UI shows.

Document
Failure

fig. 6 — a content hash drops files already seen, PII comes out before chunking, and the dashed region runs later on QStash. Each stage is a checkpoint, so a dead run can be retried from the step that broke.

The Compoze admin knowledge base page: a document table with type, size, source connector and ingest status, status filter counts above it, and a banner saying processing continues in the background. — fig. 7 — the admin view of ingest. Every document shows where it came from and where it stopped; the counts across the top are how an admin finds the failures.

Jobs are keyed by content hash, so a re-upload is a no-op; embeddings are cached by chunk hash and model, so editing one paragraph re-embeds one chunk. PII is redacted at ingest, which keeps the embeddings clean. Anything that exhausts its retries lands in a dead-letter queue with the payload and the error attached. That gave me a record of what failed and what needed retrying.

Nine tools: contextual RAG search, retrieve, summarise, read, create and update a document, web search, extract and crawl.

The Compoze agent picker showing four domain agents, each with a short description. — fig. 8 — four domain agents, each with its own prompt, tools and examples. Which ones a tenant sees is configuration.

Safety

Retrieved text is untrusted input. An instruction planted in a synced document can influence the model, so context arrives in a delimited block the prompt declares as data. Tools are scoped per agent and tenant connector. These controls limit which tools the model can call. They do not guarantee that the model will ignore a planted instruction.

Implementation inventory

domain agents: 4
agent tools: 9
models / providers: 9 / 3
knowledge bases per search: 5, in parallel
similarity floor: 0.35 cosine
test specs: 27
api routes: 53
database tables: 23
typescript: ~76k lines

Figure 09 · Chat

The answer keeps generating if the reader closes the tab

Chat streams by default. If the client disconnects before the stream completes, an abort listener hands the job to a QStash workflow. Each phase is a durable step, so a retry resumes after the last one that finished.

Question
Tokens
Hand-off

fig. 9 — chat streams by default. If the reader leaves before the answer completes, the job moves to durable QStash steps, and a retry resumes after the last one that finished.

Optimisations and operating details

Chat streams by default. If the client disconnected mid-answer and the stream had not completed, an abort listener handed the job to a QStash workflow. Each phase (init, load context, RAG query, generation, finalise) was a durable step, so a retry could resume after the last completed phase. When QStash was unreachable the route fell back to plain streaming.

Edge middleware replaced any client-supplied identity header with the user and tenant id from the verified JWT, so routes trusted it and skipped a database round trip. Indexes led with the tenant id, and chat creation was an atomic insert-on-conflict.

cursor pagination: fetch limit + 1 to learn whether a next page exists, no COUNT
adaptive polling: 3s while processing, 5s while pending, 30s idle, previous data kept so the table never flashes
connector caches: per-connector LRU with per-action TTLs for Drive and Lark, plus a sliding-window rate limit per tenant
sse anti-buffering: no-transform cache headers, X-Accel-Buffering off, framework compression disabled, per-route duration budgets on fluid compute
tool surface: only the tools the agent and the tenant connectors allow reach the model, with a tighter step budget on background runs

With no one else reviewing the code, automated checks did that job. Route contracts are typed, every payload is validated at runtime with Zod, and contract tests run on each push. Those checks let me keep changing 53 routes and 23 tables on my own. Each request is traced as one span tree with tokens, model and tenant on every span, so pricing is built on cost per tenant per answer. The latency I watched most closely was time to first token.