Semantic Chunker is a lightweight Python package for semantically-aware chunking and clustering of text. It’s designed to support retrieval-augmented generation (RAG), LLM pipelines, and knowledge processing workflows by intelligently grouping related ideas.
Traditional chunking methods for LLM pipelines often rely on fixed-size windows or naive text boundaries, which can fragment meaning, split up related ideas, or fail to capture contextual coherence. This leads to inefficient retrieval, diluted embeddings, and suboptimal responses in RAG systems. Semantic Chunker addresses this by analyzing the meaning of each chunk — using sentence embeddings and clustering — to merge semantically similar chunks into more coherent units. The result is smarter preprocessing: fewer, denser, and more relevant chunks that enhance both the performance and interpretability of your downstream models.
- Embedding-based chunk similarity (via Sentence Transformers)
- Token-aware merging with real model tokenizers
- Clustered chunk merging for optimized RAG inputs
- Preserves chunk metadata through merging
- Visual tools: attention heatmaps, semantic graphs, cluster previews
- Export options: JSON, Markdown, CSV
- CLI Interface for scripting and automation
- Debug mode with embeddings, similarity matrix, semantic pairs
pip install advanced-chunkerfrom semantic_chunker.core import SemanticChunker
chunks = [
{"text": "Artificial intelligence is a growing field."},
{"text": "Machine learning is a subset of AI."},
{"text": "Photosynthesis occurs in plants."},
{"text": "Deep learning uses neural networks."},
{"text": "Plants convert sunlight into energy."},
]
chunker = SemanticChunker(max_tokens=512)
merged_chunks = chunker.chunk(chunks)
for i, merged in enumerate(merged_chunks):
print(f"Chunk {i}:")
print(merged["text"])
print()
# Response
Merged Chunk 1
Text: Artificial intelligence is a growing field. Machine learning is a subset of AI. Deep learning uses n...
Metadata: [{'text': 'Artificial intelligence is a growing field.'}, {'text': 'Machine learning is a subset of AI.'}, {'text': 'Deep learning uses neural networks.'}]
Merged Chunk 2
Text: Photosynthesis occurs in plants. Plants convert sunlight into energy....
Metadata: [{'text': 'Photosynthesis occurs in plants.'}, {'text': 'Plants convert sunlight into energy.'}]
from semantic_chunker.visualization import plot_attention_matrix, plot_semantic_graph, preview_clusters
chunker = SemanticChunker(max_tokens=512)
debug = chunker.get_debug_info(chunks)
preview_clusters(debug["original_chunks"], debug["clusters"])
plot_attention_matrix(debug["similarity_matrix"], debug["clusters"])
plot_semantic_graph(debug["original_chunks"], debug["semantic_pairs"], debug["clusters"])chunker chunk \
--chunks path/to/chunks.json \
--threshold 0.5 \
--similarity-threshold 0.4 \
--max-tokens 512 \
--preview \
--visualize \
--export \
--export-path output/merged \
--export-format jsonExport clustered or merged chunks to:
.json: for ML/data pipelines.md: for human-readable inspection.csv: for spreadsheets or BI tools
| Argument | Description |
|---|---|
--chunks |
Path to a JSON list of text chunks (each as a dict with a text field) |
--threshold |
Agglomerative clustering distance threshold (controls granularity of clusters) |
--similarity-threshold |
Minimum cosine similarity required for two chunks to be considered semantically linked |
--max-tokens |
Maximum number of tokens allowed per merged chunk |
--preview |
Print out samples of clusters to the console |
--visualize |
Show attention matrix heatmap plot of inter-chunk similarity |
--export |
Enable file export of merged or clustered chunks |
--export-path |
Output file path prefix (e.g. output/merged) |
--export-format |
File format for export (json, csv, or md) |
| Term | Meaning |
|---|---|
| Threshold | Controls how strictly the clustering groups similar chunks. Lower = more granular clusters. |
| Similarity Threshold | Sets the minimum cosine similarity to form a semantic pair between two chunks. |
| Semantic Pairs | Pairs of chunks with high similarity; used for graph-based visualizations. |
| Attention Matrix | The full NxN matrix of similarities between every chunk embedding. |
| Merged Chunk | The final recombined text after semantic grouping. |
| Cluster | A group of semantically similar chunks found by the algorithm. |
Chunks → Embeddings → Cosine Similarity → Clustering → Merging
↓
Semantic Pairs (Optional)
↓
Visualization & Export (Optional)
Use SemanticChunkerSplitter to replace standard splitters:
from langchain_core.documents import Document
from semantic_chunker.integrations.langchain import SemanticChunkerSplitter
splitter = SemanticChunkerSplitter(cluster_threshold=0.4, return_merged=True)
docs = [Document(page_content="Some long technical text here", metadata={"source": "report.pdf"})]
split = splitter.split_documents(docs)pytest tests/Pull requests are welcome! Please open an issue first if you'd like to add a feature or fix a bug.
MIT License. See LICENSE for details.
- Sentence Transformers
- LangChain
- scikit-learn
- Hugging Face ecosystem