Chonkie Documentation — Chunking and Ingestion for RAG Pipelines
by Chonkie
Eleven chunking strategies, a written guide to picking one, and connectors straight into ten vector databases.
Overview
Chunking is the step most RAG tutorials reduce to a fixed-size slice, and it is usually the reason retrieval quality is poor. Chonkie's documentation is the most complete treatment of that step available as a library reference. It is organised as Welcome and Concepts, Getting Started (quick start, installation), and Pipelines (building a workflow, the pipeline API, configuration management), then the component categories the library is built from. Chefs parse and clean input — TextChef, MarkdownChef and TableChef, plus MistralOCR and LiteParse for document extraction. Fetchers pull files in from the local filesystem. Chunkers number eleven — Token, Fast, Sentence, Recursive, Semantic, Late, Neural, Slumber, Code, Table and TeraflopAI — and ship with an explicit guide to help you pick the right chunking strategy for a use case. Refineries post-process chunks by adding overlap or attaching embeddings. Embeddings handlers cover AutoEmbeddings, OpenAI, Azure OpenAI, Cohere, SentenceTransformer, Model2Vec, Jina, Gemini, VoyageAI and LiteLLM, with custom handler support. Handshakes write chunks straight into Chroma, Elasticsearch, LanceDB, Milvus, MongoDB, pgvector, Pinecone, Qdrant, Turbopuffer or Weaviate, and Porters export to JSON or Hugging Face Datasets. There is also a REST API server with Docker deployment, a CLI, a visualizer and logging utilities. The library is MIT-licensed with 4.7k GitHub stars and roughly 2,200 commits, installs with pip install chonkie or pip install "chonkie[all]", keeps the base wheel near 505 KB, and offers JavaScript as well as Python bindings.
At a Glance
- Topic
- RAG
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- ~15 minutes to a working chunker; ~1-2 hours for the full chunker and pipeline documentation
- Provider
- Chonkie
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Choose between eleven chunking strategies using the documentation's own decision guide rather than guessing
- ✓Understand how semantic, late and neural chunking differ from naive fixed-size token splitting
- ✓Parse markdown, tables and scanned documents into clean text before chunking them
- ✓Add overlap or precomputed embeddings to chunks with refineries to improve recall
- ✓Write chunks straight into Chroma, Qdrant, pgvector, Milvus and seven other stores via handshakes
- ✓Compose fetch, parse, chunk, refine and store into one reproducible declarative pipeline
- ✓Swap embedding providers — OpenAI, Cohere, Jina, VoyageAI, SentenceTransformers — behind a single interface
Highlights
- •Eleven named chunkers with a written guide to choosing between them; most libraries ship two and no guidance
- •Handshakes remove the glue code between chunker and vector database for ten different stores
- •MIT-licensed with a base wheel near 505 KB, light enough to drop into an existing pipeline
- •Dedicated Code and Table chunkers handle the two content types generic splitters mangle worst
- •4.7k GitHub stars and active development across docs, tests and code, plus JavaScript bindings
Who It’s For
Best For
- ✓RAG engineers whose retrieval quality is poor and who suspect chunking is the cause
- ✓Teams building document ingestion over mixed markdown, code and tabular sources
- ✓Anyone who has written the same parse-chunk-embed-upsert glue code more than twice
- ✓Engineers benchmarking semantic versus recursive versus late chunking on their own corpus
Prerequisites
- •Working Python plus familiarity with embeddings and vector search basics
- •An existing or planned vector database — the handshakes assume somewhere to write to
- •Helpful: having already built one naive RAG pipeline, so the failure modes are recognisable
FAQ
What is Chonkie Documentation — Chunking and Ingestion for RAG Pipelines?
The official documentation for Chonkie, an MIT-licensed Python library that handles the ingestion half of a RAG pipeline: parsing, chunking, embedding, refining and writing to a vector store. It documents eleven chunkers — token, fast, sentence, recursive, semantic, late, neural, slumber, code, table and TeraflopAI — with a guide to choosing between them. Most RAG tutorials hand-wave chunking; this is the part they skip.
Is Chonkie Documentation — Chunking and Ingestion for RAG Pipelines free?
Chonkie Documentation — Chunking and Ingestion for RAG Pipelines is free to access.
What level is Chonkie Documentation — Chunking and Ingestion for RAG Pipelines for?
Chonkie Documentation — Chunking and Ingestion for RAG Pipelines is aimed at a intermediate audience. Recommended background: Working Python plus familiarity with embeddings and vector search basics, An existing or planned vector database — the handshakes assume somewhere to write to, Helpful: having already built one naive RAG pipeline, so the failure modes are recognisable.
How long does Chonkie Documentation — Chunking and Ingestion for RAG Pipelines take?
Expect roughly ~15 minutes to a working chunker; ~1-2 hours for the full chunker and pipeline documentation. Most learners work through it at their own pace.
What will I learn from Chonkie Documentation — Chunking and Ingestion for RAG Pipelines?
You'll learn: Choose between eleven chunking strategies using the documentation's own decision guide rather than guessing; Understand how semantic, late and neural chunking differ from naive fixed-size token splitting; Parse markdown, tables and scanned documents into clean text before chunking them; Add overlap or precomputed embeddings to chunks with refineries to improve recall; Write chunks straight into Chroma, Qdrant, pgvector, Milvus and seven other stores via handshakes; Compose fetch, parse, chunk, refine and store into one reproducible declarative pipeline; Swap embedding providers — OpenAI, Cohere, Jina, VoyageAI, SentenceTransformers — behind a single interface.
Topics
Sources
This page was written from 3 sources, 1 on domains other than docs.chonkie.ai.