AI
Retrieval-Augmented Generation in Production: What Changes After the Demo
A RAG prototype can be built in an afternoon. Making it accurate, secure and dependable takes engineering. These are the areas that matter most.
Retrieval-augmented generation (RAG) grounds a language model's answers in your own documents. It is one of the most practical ways to apply generative AI in a business. But the gap between a convincing demo and a system people trust every day is significant.
Ingestion is a data engineering problem
Answer quality is bounded by the quality of what you index. Production pipelines need to handle many document formats, extract tables and structure, remove duplicates, track versions and re-index when sources change. Treat ingestion as a pipeline with tests and monitoring, not a one-off script.
Retrieval quality deserves the most attention
- Chunk documents along natural boundaries such as headings and sections, preserving context.
- Combine semantic vector search with keyword search — hybrid retrieval handles names, codes and acronyms better.
- Use a re-ranking step to promote the most relevant passages.
- Attach metadata such as product, region and date to enable filtering.
Respect permissions
An assistant must never reveal content a user could not otherwise access. Enforce authorisation at retrieval time by filtering on access-control metadata synchronised from source systems, rather than relying on the model to withhold information.
Design for transparency
Show citations so users can verify answers, and make it acceptable for the system to say it does not know. Clear source attribution builds trust faster than fluent but unverifiable responses.
Operate it like any other production system
Monitor latency, cost per query, retrieval hit rates and user feedback. Log interactions in line with your privacy policy so you can investigate issues and continuously improve the system.