Chunking documents without losing meaning
Split a document into useful pieces while keeping its headings, tables, source pages and permissions. Learn when invoice data should stay in rows, how to split policies, and how to check the result.
The question: How do we split documents so search finds a complete answer instead of a broken sentence?
You’ll learn: Design chunks that preserve policy rules and table meaning, then compare them using questions with known evidence.
For: People building or reviewing a document search system; no database background required.
Illustrative worked example · AI-assisted source check . This is not a human expert endorsement.
What is a chunk?#
A chunk is a piece of a document that your search system can find on its own. It might contain one policy rule, a few related paragraphs, or a table with its heading. Chunking means choosing where those pieces begin and end.
Suppose a policy says, “Price differences need buyer approval.” The next sentence says, “This also applies when the total invoice amount is below the usual approval limit.” A split between those sentences can hide the exception. A useful chunk keeps the rule and its limit together.
Small chunks make specific passages easier to find. Larger chunks give more surrounding information. Neither size wins every time. The job is to return enough evidence to answer the question without surrounding it with unrelated pages. Document structure is a useful starting point.
Primary guidance on fixed sizes, document structure, overlap and model input limits. The settings proposed below are our starting experiment, not a vendor rule.
A policy passage becomes a searchable chunk. Its source record stays attached, so a reader can inspect the same rule in the original document.
Select a box to inspect its role. On a small screen, scroll the diagram sideways.A versioned PDF with headings, paragraphs, tables and page locations.
Read the complete diagram as text
- Original policy: A versioned PDF with headings, paragraphs, tables and page locations.
- Read the structure: Identify a heading and the clauses that belong to it. Keep qualifications and table headers.
- Searchable passage: A complete rule with a heading path. Split a long continuation only when needed.
- Source and permissions: Document ID, version, pages, organization, permitted readers and effective dates stay attached.
- Original policy → extract → Read the structure
- Read the structure → split at a useful boundary → Searchable passage
- Original policy → preserve source details → Source and permissions
- Searchable passage → link back → Source and permissions
First decide what needs to be a record#
An invoice is not just prose. Its invoice number, supplier, purchase order, currency, amounts and line items should become fields in records. You compare those fields with the enterprise resource planning system, or ERP. You should not search for a vaguely similar number to decide whether two invoices match.
Keep each invoice line with its line number, description, quantity, unit, price and source location. A line that continues on another PDF page is still the same business record. Check the extraction before doing calculations. Chunking cannot repair a decimal point that was read incorrectly.
| Document content | How to keep it | Why |
|---|---|---|
| Invoice number and totals | Typed fields, with the original text and page location | Match identifiers and calculate amounts exactly. |
| Invoice lines | One record per business line, with column names | Keep quantities tied to the correct item and unit. |
| Approval policy | Heading plus its rule and qualifications | Find the instruction that applies to the exception. |
| Long explanation in a manual | Related paragraphs with a link to the full section | Find a short passage, then read more when needed. |
The PDF extraction guide shows the stage before this one. Start there if the PDF has scans, difficult tables or missing text.
A practical starting rule for policies#
Start a new chunk at a section or clause boundary. Attach the document title and heading path. Keep a rule with nearby conditions such as “unless,” “except” and “only when.” These words often change what a person is allowed to do.
For the invoice teaching example, try a target cap of about 400 tokens, including the added heading. A token is a small unit of text used by a model; it is not always a whole word. Count with the tokenizer for your chosen embedding model. This is a setting to test, not a universal best size.
If a clause is too long, split at a sentence or paragraph boundary and repeat up to 50 tokens from the previous piece only where the same clause continues. Repeating text is called overlap. Do not copy the end of one unrelated section into the next. Keep the full clause as a linked parent record.
A model’s maximum input length is a hard limit. Your chunk size is a search design choice. A model accepting a long document does not mean one embedding will help you find every small rule inside it. Include headings and any model-specific prefixes when checking the input length.
Primary explanation of headings, paragraph structure and parent-child fields. We use these concepts without requiring this service.
Keep table headers and units#
The following invented policy table applies to one company. It is teaching data, not an accounting rule. Imagine the second row falls onto the next PDF page. A loose chunk containing only “Freight | Buyer review” would lose which question the table answers.
Document: Invoice review policy
Section: 4 / Extra charges
Version: 3
Pages: 6–7
Charge type | Required evidence | Next step
Freight | Agreed order term | Buyer review if absent
Packing | Agreed order term | Buyer review if absentKeep this small table as one chunk. For a long table, group complete rows and repeat the column headers and section heading in each group. Preserve notes that change how those rows are read. Do not split a numeric cell away from its currency, unit or row label.
Store source details separately from the text used for search. A useful record can look like this:
{
"chunk_id": "policy-v3-extra-charges-01",
"document_id": "invoice-review-policy",
"document_version": 3,
"parent_section_id": "extra-charges",
"heading_path": [
"Invoice review policy",
"Extra charges"
],
"pages": [
6,
7
],
"organization_id": "example-company",
"permission_group": "accounts-payable",
"valid_from": "2026-07-01",
"valid_to": null,
"text": "Charge type | Required evidence | Next step\nFreight | Agreed order term | Buyer review if absent"
}An application must check those permissions; writing a group name in the record does not enforce access. Version and validity answer different questions. Version identifies the source you used. Effective dates tell you when the rule applies. A newly uploaded file may describe an old period.
Find a passage, then bring back enough context#
Use word search for names and exact phrases. Add vector search when a reader uses different wording for the same idea. An embedding is a list of numbers used to compare meaning; the embeddings article explains how it is made.
You can search a small child chunk and then fetch its larger parent section. This gives a focused match with the surrounding explanation. Check the parent’s access and version too. Set a limit on the total text returned, so one match does not pull an entire handbook into the answer.
When adjacent chunks repeat an overlap, remove that repeated passage before showing the evidence. Otherwise, two copies of one rule can look like two supporting sources. Keep the original chunk IDs so the result is still traceable.
Check the evidence before judging the answer#
Ask a reviewer to write questions and mark the source passages needed for each answer. Include a rule with an exception, a table across pages, an old policy, an unknown answer and a document belonging to another company. Keep some questions aside while you tune the settings. These held-out questions check whether the changes help beyond the examples you already studied.
Compare whole sections, heading-based chunks near 400 tokens, and smaller chunks near 200 tokens. Keep the same documents, embedding model, search method and result count for each run. Then inspect the returned evidence. Do not change everything together or you will not know what helped.
| Check | What to record |
|---|---|
| Did we find the rule? | Whether the first five results contain the required passage and its qualification. |
| Can someone verify it? | Whether page, heading and version lead to the correct source. |
| Did anything break? | Missing units, detached table headers, duplicates or chopped conditions. |
| Did access work? | Zero results from documents the reader cannot access. |
| What did it cost? | Index size, embedding input tokens, search time and text sent for answering. |
Choose the smallest approach that reliably returns complete, permitted evidence on the held-out questions. If it misses a rule, inspect the extraction and chunk boundary before buying a larger model. For an end-to-end build, continue to Build a RAG application you can inspect.