Retrieval quality is decided before the first query. Preprocessing turns messy PDFs and HTML into clean text. Chunking cuts that text into units you embed and fetch. Bad chunks—too huge, too tiny, or split mid-table—make even a perfect vector database look broken.
An embedding is one point summarizing a chunk. A 40-page handbook in one chunk becomes oatmeal: every topic blended. Three words alone lack context. You want chunks that are one coherent idea, large enough to answer something, small enough to stay focused.
Overlap exists because answers often straddle boundaries. A sentence at the end of chunk 4 may need the first lines of chunk 5.
source_id, title, url, updated_at, section_path, access tags.| Document type | Suggested approach | Why |
|---|---|---|
| PDF plain prose | pypdf | Fast when the file is mostly text |
| PDF with tables | pdfplumber or Camelot | Preserves row/column structure |
| Scanned / image PDF | OCR (optical character recognition) | No text layer to extract otherwise |
| Complex layout | layout-aware parser (unstructured, docling) | Reading order matters |
| HTML pages | BeautifulSoup, trafilatura | Strip boilerplate; keep useful content |
OCR turns scanned images of text into machine-readable text. If a PDF is just an image, a plain extractor may return empty strings—nothing to embed.
Quick probe before parsing:
from pypdf import PdfReader
def pdf_has_text_layer(path, sample=3):
reader = PdfReader(path)
return any((p.extract_text() or "").strip() for p in reader.pages[:sample])| Strategy | How it works | Best when |
|---|---|---|
| Fixed-size | Split by token count with overlap | Baseline or uniform prose |
| Sliding window | Move a window with 10–20% overlap | Answers cross boundaries |
| Structure-aware | Split on headings, clauses, sections | Markdown, legal text, code, API docs |
| Semantic chunking | Split when topic similarity drops | Topic-shifting docs; quality over cost |
| Recursive splitting | Try structure first, then smaller pieces | General-purpose practical choice |
| Parent–child | Index small child chunks; return larger parent for context | Balance precision and completeness |
Common starting point: about 512 tokens with 10–20% overlap. Match chunk size to the kind of answer you expect—small for FAQ facts, larger for narrative or code.
If the knowledge base is tiny and already fits in the model's context window, you may get a simpler result by putting the text directly in the prompt instead of building a retrieval system.
Structure-aware splitting over Markdown headings with word-based overlap.
import re
from dataclasses import dataclass
@dataclass
class Chunk:
id: str
text: str
section: str
source: str
def split_markdown(text: str, source: str, max_words: int = 80, overlap: int = 15) -> list[Chunk]:
parts = re.split(r"(?m)^(#{1,3} .+)$", text)
sections: list[tuple[str, str]] = []
current_title = "intro"
buf = []
for part in parts:
if re.match(r"^#{1,3} ", part or ""):
if buf:
sections.append((current_title, " ".join(buf).strip()))
buf = []
current_title = part.lstrip("#").strip()
elif part and part.strip():
buf.append(part.strip())
if buf:
sections.append((current_title, " ".join(buf).strip()))
chunks: list[Chunk] = []
n = 0
for title, body in sections:
words = body.split()
if not words:
continue
start = 0
while start < len(words):
end = min(start + max_words, len(words))
piece = " ".join(words[start:end])
chunks.append(Chunk(f"{source}_{n}", piece, title, source))
n += 1
if end == len(words):
break
start = max(0, end - overlap)
return chunksProduction code usually counts real tokenizer tokens, not words.
source or access tags cannot be cited or filtered safely.Preprocess to clean, structured text, then chunk with coherent boundaries, sensible size, overlap, and metadata—because embeddings can only retrieve what you gave them.