Loading and chunking

Turning PDFs, web pages and other sources into clean text with metadata, then into chunks sized for retrieval.

Document loaders

Also called readers, connectors

What: Components that read a source (files, web pages, Notion, databases) and return document objects holding text plus metadata such as file name and page.
Why: Hundreds of ready-made loaders save writing parsers, and every document arrives in one shape the rest of the pipeline understands.

Python

from langchain_community.document_loaders import (
    PyPDFLoader, WebBaseLoader,
)

docs = PyPDFLoader("handbook.pdf").load()
docs += WebBaseLoader("https://example.com/faq").load()
print(docs[0].page_content[:200])
print(docs[0].metadata)

Metadata

What: Key-value facts attached to each document and inherited by its chunks: source, title, section, date, owner team, access level.
Why: Metadata powers filtering ("only billing docs"), citations and access control, and can be added to the text that gets embedded for better matches.

Watch out

Dropping metadata at load time; adding it later means re-ingesting everything.

Chunking

Also called text splitting, node parsing

What: Splitting documents into passages small enough to embed and retrieve precisely, ideally along natural boundaries such as paragraphs and sentences.
Why: A chunk should hold one idea: too large and the match is diluted, too small and it lacks the context to be useful.

Python

from langchain_text_splitters import (
    RecursiveCharacterTextSplitter,
)

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000, chunk_overlap=150, add_start_index=True
)
chunks = splitter.split_documents(docs)

Chunk size and overlap

What: Size is the maximum length of a chunk; overlap repeats the end of one chunk at the start of the next. LangChain's recursive splitter counts characters by default, LlamaIndex's sentence splitter counts tokens.
Why: Overlap stops an answer being cut in half at a boundary. Tune size on your own questions rather than copying defaults.

Watch out

Comparing sizes across tools without checking whether they mean characters or tokens.

Structure-aware splitting

Also called Markdown splitting, semantic chunking

What: Splitting on the document's own structure (headings, code blocks, HTML sections) or on shifts in meaning, and keeping the heading path as metadata.
Why: A chunk that knows it sits under "Refunds > Sale items" retrieves and reads far better than an anonymous paragraph.

Parsing difficult documents

What: PDFs with tables, columns, scans and charts lose structure when read as plain text. Layout-aware parsers or OCR recover reading order and tables, often as Markdown.
Why: Garbage in, garbage out: if a table is flattened into a jumble of numbers, no retriever can fix it.

Watch out

Never looking at the parsed text; always read a sample of chunks before tuning anything else.

Incremental ingestion

Also called deduplication, upserts

What: Giving each document a stable id and a content hash, so re-running ingestion only adds new documents, updates changed ones and deletes removed ones.
Why: Re-embedding everything nightly is slow and costly, and duplicate chunks crowd out other results.

Go deeper