Checklist · AI
RAG Production Checklist
Opens with creditsA checklist for taking retrieval-augmented generation from demo to production. It covers ingestion and chunking, document permissions and injection, embeddings and indexes, hybrid search and reranking, grounded answers with citations, retrieval and answer evals, freshness, monitoring and cost, with a severity and a verification step for each check.
Written by Tauseef Fayyaz7 chapters, 30 checksAbout 15 minReviewed 12 October 2026What you will learn
7 chapters · about 15 min
1Data ingestion and chunkingWhat goes into the index decides what can come out. These checks make sure documents are parsed cleanly and split into chunks that make sense on their own.5 checks · 2 min
- 1Extract clean text, tables and structure from every source format
- 2Split on document structure, not a fixed character count alone
- 3Give each chunk the context it needs to be understood alone
- 4Remove duplicates, drafts and junk before indexing
- 5Record every document that fails to ingest
5 in this chapterOpen this chapter2Access control and document securityRetrieval puts your documents into prompts, so it inherits every permission and trust problem those documents have. These checks are about who can see what, and what a document can make the model do.5 checks · 3 min
- 1Filter retrieval by the current user's permissions on every query
- 2Propagate permission changes and deletions to the index quickly
- 3Keep each customer's data in its own namespace or index
- 4Treat retrieved text as untrusted data, never as instructions
- 5Index only sources you trust and record where each chunk came from
5 in this chapterOpen this chapter- Embeddings and index choiceThe embedding model and the vector index set the ceiling on recall and the floor on latency. Choose both with your own data, then pin them.4 checks · 2 minOpens with credits
- Metadata, hybrid search and rerankingPure vector search misses exact terms and ignores structure. Metadata filters, keyword search and a reranker fix most of what users notice.4 checks · 2 minOpens with credits
- Grounding and citationsRetrieval only helps if the answer actually uses it. These checks keep answers tied to sources users can open and check.4 checks · 2 minOpens with credits
- Evaluation of retrieval and answersEvaluate retrieval and generation separately, so you know which half to fix. Then run both on every change.4 checks · 2 minOpens with credits
- Freshness, monitoring and costA RAG system is a data pipeline that never stops. These checks keep the index current, the answers watched and the bill predictable.4 checks · 2 minOpens with credits
About this checklist
A RAG demo takes an afternoon: split some documents, embed them, retrieve the top five chunks and ask a model to answer. A RAG system in production is different. It has to find the right passage among millions, respect who is allowed to see what, cite its sources, stay current when documents change, and not follow instructions someone hid in a PDF. This RAG production checklist covers what to check before you launch and what to keep checking after.
Most RAG failures are retrieval failures. If the right chunk never reaches the model, no prompt can fix the answer. That is why the checklist starts with how documents are ingested and chunked, and why evaluation measures retrieval separately from the final answer. Anthropic's contextual retrieval experiments are a good example of how much retrieval work pays off: adding context to chunks, combining embeddings with keyword search and reranking cut their retrieval failure rate from 5.7% to 1.9%.
Security gets its own chapter early. Vector and embedding weaknesses are on the OWASP Top 10 for LLM Applications, and the most common real problem is simple: a retrieval index that ignores document permissions will happily show one user another user's files.
Each check has a severity (must, should or nice) and a concrete way to verify it, such as a query to run, a metric to read, or a test to add to your eval set.
Common questions
What is RAG in simple terms?
Retrieval-augmented generation means searching your own documents for passages relevant to a question and giving those passages to a language model to answer from. It lets the model use current, private information without retraining, and lets you show sources for each answer.
What is the best chunk size for RAG?
There is no single best size. Split on document structure such as headings and paragraphs, keep chunks to a few hundred tokens, and add a little overlap. Then test two or three sizes against your own retrieval eval set and keep the one with the best recall.
What is the difference between hybrid search and vector search?
Vector search finds passages with similar meaning, while keyword search such as BM25 finds exact terms like error codes and names. Hybrid search runs both and merges the results, often with reciprocal rank fusion, so you get the strengths of each.
How do I evaluate a RAG system?
Evaluate retrieval and answers separately. For retrieval, measure recall at k and context precision on labelled questions. For answers, measure faithfulness to the retrieved context and relevance to the question, using code checks and a model grader you have spot-checked by hand.
How do I stop RAG from leaking documents users should not see?
Store access control information with every chunk and filter by the current user's permissions inside the retrieval query. Sync permission changes and deletions quickly, keep tenants in separate namespaces, and test with two users who have different access.
Sources (7)
Checked against these, last on 12 October 2026.
- Introducing Contextual Retrieval, Anthropicwww.anthropic.com/news/contextual-retrieval
- LLM08:2025 Vector and Embedding Weaknesses, OWASP GenAI Security Projectgenai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/
- OWASP Top 10 for LLM Applications 2026, OWASP GenAI Security Projectgenai.owasp.org/resource/owasp-genai-llm-top-10-2026/
- LLM Prompt Injection Prevention Cheat Sheet, OWASP Cheat Sheet Seriescheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html
- pgvector, pgvector on GitHubgithub.com/pgvector/pgvector
- Reciprocal rank fusion, Elastic docswww.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion
- Faithfulness, Ragas docsdocs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/
Related on Sefism
Picked from the other libraries by shared topics.
Roadmaps
See allCheat sheets
See allBest practices
See all- Best Practices for Building LLM AppsHow to build LLM apps and agents that hold up in production: prompts, evals, prompt injection, tool permissions, cost, logging and privacy.
- Prompt Engineering Best PracticesHow to write prompts that work the same way on the hundredth input as on the first: clear instructions, structure, examples, reasoning, output control and testing.
- Best Practices for AI Coding AssistantsHow to get reliable code from AI coding assistants and agents: instruction files, task prompts, verification, review, security, dependencies and session hygiene.
Checklists
See all- LLM Evaluation ChecklistAn LLM evaluation checklist: success criteria, eval datasets, code and model graders, judge calibration, agent evals, safety tests and production monitoring, with how to verify each check.
- AI Feature Launch ChecklistAn AI feature launch checklist: use case fit, evals, UX for errors, disclosure, human override, abuse filters, rate limits, cost caps, prompt privacy, monitoring and rollback.
- EU AI Act Readiness ChecklistAn EU AI Act readiness checklist for teams building or deploying AI: scope and roles, prohibited practices, risk classification, transparency, GPAI duties, oversight and dates.
Project structures
See all- GenAI RAG app (FastAPI) project structureFor developers turning a "chat with your documents" demo into an app people can rely on. A FastAPI backend where every layer has a home: ingestion workers, retrieval, prompts as versioned files, model clients with fallbacks and cost caps, guards, evals that gate CI, and tracing, so a wrong answer can be traced to the step that caused it.
- Machine learning project project structureFor students and analysts whose model works in a notebook and now needs to be reproducible, reviewable, served and watched. It starts from the Cookiecutter Data Science v2 layout (immutable raw data, numbered notebooks, a real Python package) and grows into versioned data with DVC, typed parameters, data validation with pandera, experiment tracking and a model registry with MLflow, a FastAPI prediction service, drift monitoring and, for a team, scheduled retraining, model cards, fairness checks and shadow deployment.
- AI agent with an MCP server project structureFor developers who want to give AI assistants safe access to their own data or API. A TypeScript MCP server built on the official SDK, organised by feature (tasks, projects) with each feature owning its tools, resources and prompts, served over stdio for local hosts and Streamable HTTP with OAuth for remote ones, plus a small agent and the evals that prove a model can use it.
Career questions
See all- Which AI tools do employers expect junior developers to know in 2026?Employers rarely require a specific brand. They expect you to use an AI coding assistant well (in the editor and in the terminal), to review and test what it produces, to call an LLM API in a small feature, and to explain when you would not use AI. Pick one assistant, learn it properly, and show judgement in your projects.
- How do I use AI to become a better engineer instead of becoming dependent on it?Decide the job AI does before you open it. Use it as a tutor, reviewer and sparring partner that explains and challenges, not a vending machine that hands you finished code. Attempt first, ask second, and always be able to explain and defend what you ship.
- How do I start learning AI agents as a software engineer?Start with one LLM API and plain code: a loop that calls the model, lets it pick a tool, runs the tool and feeds the result back. Build one small agent that does a real task end to end, add evaluation and guardrails, and only then reach for a framework. Agents are software engineering first.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com