Chapter 1 of 7 · 5 checks · 2 min
Data ingestion and chunking
What goes into the index decides what can come out. These checks make sure documents are parsed cleanly and split into chunks that make sense on their own.
Extract clean text, tables and structure from every source formatMust
Why it matters and how to check
PDFs, slides, HTML and scanned files often extract as broken text: headers and footers repeated on every page, tables flattened into a wall of numbers, two columns interleaved, or nothing at all from an image. Garbage in the index means garbage retrieved.
Use a parser suited to each format, strip navigation and boilerplate, keep headings, and run OCR on scanned pages. To verify, pick 20 random documents across formats and compare the stored text with the original side by side. Count documents with missing sections, mangled tables or empty text, and fix the parser before indexing more.
Split on document structure, not a fixed character count aloneMust
Why it matters and how to check
Cutting every 1,000 characters splits sentences, separates a heading from its paragraph and breaks tables in half. Split on headings, sections and paragraphs first, then apply a size limit, with a small overlap so ideas that cross a boundary are not lost. Anthropic's retrieval write-up notes that chunk size, boundaries and overlap all affect retrieval performance, and that chunks are usually no more than a few hundred tokens.
To verify, print 30 random chunks. Each should be readable on its own, start at a sensible boundary and not end mid-sentence.
Give each chunk the context it needs to be understood aloneShould
Why it matters and how to check
A chunk that says "revenue grew 3% over the previous quarter" is useless without the company and the quarter. Prepend the document title and section path, or a short generated summary of where the chunk sits, before embedding and keyword indexing. Anthropic reported that contextual embeddings alone cut retrieval failures by 35%, and combined with contextual BM25 by 49%.
To verify, run your retrieval eval with and without the added context and compare recall at your chosen top k. Keep the change only if recall improves on your own data.
Remove duplicates, drafts and junk before indexingShould
Why it matters and how to check
Duplicate pages, old drafts, auto-generated files and near-identical versions crowd the top results with the same text and push out the passage that answers the question. Decide which sources are authoritative, skip the rest, and deduplicate by content hash or near-duplicate detection.
To verify, count chunks that share an identical hash, and sample your top 10 results for common queries. If several results come from copies of the same document, deduplication is not working.
Record every document that fails to ingestShould
Why it matters and how to check
Silent ingestion failures create blind spots: a policy that never made it into the index looks, to users, like the assistant does not know the answer. Log each document's status, parser used, chunk count and error, and alert when the failure rate jumps.
To verify, compare the number of source documents with the number of indexed documents per source. The gap should be explained by a list of known, logged failures, not by silence.