Voice agent knowledge systems with Bolna

Bolna builds voice agents that hold real phone conversations, and they wanted those agents to answer from a knowledge base during a live call. That sounds like a standard RAG feature until you remember where it has to run: inside a speech-to-text, LLM, text-to-speech pipeline where every pause is audible.

On a chat screen, a second of retrieval is invisible. On a phone call it is dead air, and callers hear dead air as a dropped line. So the real job was not “add retrieval”. It was adding retrieval without the caller ever noticing it was there.

Why a Pause Is a Bug in Voice

A voice agent only sounds natural if the pipeline keeps moving. Speech recognition feeds the LLM, and the LLM’s tokens stream into speech synthesis. Any long operation in that chain, like a blocking database read or a retrieval step that waits for a full transcript, stops the whole line.

The knowledge base brought the hardest kind of work into that chain: retrieval that had to happen while the person on the other end was still speaking. Bolna’s agents needed to handle complex customer queries, and the latency of looking things up could not leak into the conversation.

Keeping Retrieval off the Critical Path

We built the knowledge base on LanceDB as the vector store and designed it around asynchronous, queue-based streaming rather than request-and-response calls. A request-response design forces each stage to wait for the one before it, so we let the stages overlap instead.

Speech-to-text output is buffered rather than awaited as a complete turn, which means the LLM does not sit idle until the caller finishes. The LLM then processes the query with RAG retrieval running in parallel, and tokens stream to speech synthesis as they are generated. Audio playback starts before the model has finished its answer.

Everything runs on non-blocking I/O and concurrent processing. That is what keeps a slow lookup from ever holding up the voice.

A Consistent Snapshot of the Knowledge

Knowledge files change: a client uploads a new policy document, or edits an old one. If retrieval read files mid-update, an agent could quote half of one version and half of another.

So we put a proxy engine in front of the knowledge files. It handles file management and versioning, and the retrieval side always reads a consistent snapshot.

What Bolna Got

• An asynchronous, queue-based architecture for contextual responses during live calls.
• Buffered token streaming with concurrent RAG lookups.
• No blocking I/O left to interrupt conversational flow.
• A production-ready knowledge-base system for high-volume call traffic.

Under the hood it is LanceDB, Python, RAG and vector retrieval, held together by async I/O and a streaming architecture.