Retrieval-augmented generation (RAG) looks simple in a notebook: embed documents, store vectors, attach a prompt, call a model. Production is a different beast—latency budgets, stale content, authorization boundaries, and evaluation loops dominate the engineering calendar.
Chunking is the first lever. Semantic chunks usually outperform arbitrary token windows for factual recall, but they require investment in cleaning HTML/PDF noise and preserving headings and tables. Hybrid retrieval (BM25 + vectors) still wins many enterprise corpora where keyword overlap matters as much as semantic similarity.
Evaluation cannot be an afterthought. You need labeled question-answer pairs from real users, automatic regression suites on golden datasets, and online checks for toxicity, PII leakage, and citation faithfulness. Without these, teams chase anecdotal bugs while the model silently drifts as the knowledge base changes.
Latency and cost follow from architecture: cache embeddings for stable corpora, stream tokens to the UI, batch where possible, and cap context windows deliberately. Observability should include retrieval traces—what chunks were selected, with what scores—so incidents are debuggable without reproducing user sessions by hand.
Human-in-the-loop remains essential for regulated domains or high-stakes answers. Design explicit escalation paths, queue review tooling, and feedback capture that feeds back into chunk metadata and evaluation sets.
Finally, treat RAG as a data product. Owners, SLAs, and change management for the knowledge base matter more than the embedding model du jour. If your content pipeline is messy, RAG will amplify the mess—fix ingestion and metadata before chasing marginal recall gains.
Related: see AI strategy & MLOps, anonymized retail copilot case patterns, and more resources.