Loading and chunking

Turning PDFs, web pages and other sources into clean text with metadata, then into chunks sized for retrieval.

Document loaders

Also called readers, connectors

What: Components that read a source (files, web pages, Notion, databases) and return document objects holding text plus metadata such as file name and page.
Why: Hundreds of ready-made loaders save writing parsers, and every document arrives in one shape the rest of the pipeline understands.

Python

from llama_index.core import SimpleDirectoryReader

docs = SimpleDirectoryReader(
    "data", required_exts=[".pdf", ".md"], recursive=True
).load_data()
print(docs[0].text[:200])
print(docs[0].metadata)

Metadata

What: Key-value facts attached to each document and inherited by its chunks: source, title, section, date, owner team, access level.
Why: Metadata powers filtering ("only billing docs"), citations and access control, and can be added to the text that gets embedded for better matches.

Watch out

Dropping metadata at load time; adding it later means re-ingesting everything.

Chunking

Also called text splitting, node parsing

What: Splitting documents into passages small enough to embed and retrieve precisely, ideally along natural boundaries such as paragraphs and sentences.
Why: A chunk should hold one idea: too large and the match is diluted, too small and it lacks the context to be useful.

Python

from llama_index.core.node_parser import SentenceSplitter

splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64)
nodes = splitter.get_nodes_from_documents(docs)
print(len(nodes), nodes[0].get_content()[:200])

Chunk size and overlap

What: Size is the maximum length of a chunk; overlap repeats the end of one chunk at the start of the next. LangChain's recursive splitter counts characters by default, LlamaIndex's sentence splitter counts tokens.
Why: Overlap stops an answer being cut in half at a boundary. Tune size on your own questions rather than copying defaults.

Watch out

Comparing sizes across tools without checking whether they mean characters or tokens.

Structure-aware splitting

Also called Markdown splitting, semantic chunking

What: Splitting on the document's own structure (headings, code blocks, HTML sections) or on shifts in meaning, and keeping the heading path as metadata.
Why: A chunk that knows it sits under "Refunds > Sale items" retrieves and reads far better than an anonymous paragraph.

Parsing difficult documents

What: PDFs with tables, columns, scans and charts lose structure when read as plain text. Layout-aware parsers or OCR recover reading order and tables, often as Markdown.
Why: Garbage in, garbage out: if a table is flattened into a jumble of numbers, no retriever can fix it.

Watch out

Never looking at the parsed text; always read a sample of chunks before tuning anything else.

LlamaParse

What: LlamaIndex's hosted document parsing service, part of LlamaCloud, which turns complex PDFs, slides and spreadsheets into clean Markdown or structured output for ingestion.
Why: It handles tables and multi-column layouts that local text extraction mangles, at the cost of sending documents to an external service.

Watch out

Sending confidential documents to a hosted parser without checking your data handling rules first.

Incremental ingestion

Also called deduplication, upserts

What: Giving each document a stable id and a content hash, so re-running ingestion only adds new documents, updates changed ones and deletes removed ones.
Why: Re-embedding everything nightly is slow and costly, and duplicate chunks crowd out other results.

Go deeper