All storiesID
tutorialRAGAITypeScriptPython

Building RAG in Production: What I've Learned

Retrieval-Augmented Generation sounds simple in tutorials. Here's what actually happens when you ship it to real users.

2 min read

RAG tutorials make it look easy: chunk documents, embed them, store in a vector DB, retrieve on query, pass to LLM. Done.

What they don't tell you is what breaks at scale.

Chunking Is Not One-Size-Fits-All

The standard advice is "chunk your documents into 512-token blocks." That works great for Wikipedia articles. It falls apart for technical documentation, contracts, or anything with structure.

What I've learned:

  • Semantic chunking (split on meaning, not token count) produces dramatically better retrieval
  • Overlapping chunks reduce the chance of losing context at boundaries
  • For structured documents (PDFs, HTML), preserve the hierarchy — a heading tells you a lot about what follows it

Embedding Model Choice Matters More Than You Think

text-embedding-ada-002 is fine. But it's not optimal for domain-specific content. If you're building a legal or medical RAG system, a domain-tuned embedding model will outperform a general one significantly.

The evaluation is straightforward: run a set of ground-truth queries, measure recall@k. Pick the model that retrieves the right documents most often.

The Retrieval Layer Is Where Products Win or Lose

Most tutorials stop at vector similarity search. Real RAG systems need:

  1. Hybrid retrieval — vector search + BM25 keyword search, reranked together
  2. Metadata filtering — restrict retrieval to documents relevant to the current user or context
  3. Query rewriting — transform the user's question into a form better suited for retrieval

I use Qdrant in production. Its filtering + payload indexing makes metadata-aware retrieval clean to implement.

Observability Is Non-Negotiable

The hardest part of debugging a RAG system is that failures are silent. The LLM always produces an answer — it's just wrong.

Build in logging from day one:

  • What was retrieved for each query?
  • What was the similarity score?
  • Did the retrieved docs actually contain the answer?

Without this, you're flying blind.

The Payoff

When it works — when the system retrieves the exact right context and the LLM synthesizes it into a clear answer — it feels like the product actually understands the user.

That feeling is worth the engineering.