**Chunklet-py: One Library to Split Them All - Sentence, Code, Docs**

🔧 What It Does ## 🚀 Key Features ## 💻 Quick Example ## 📊 Why It Matters ## 🛠️ Installation ## 📈 Community & Stats I've been working on Chunklet-py - a powerful Python library for intelligent text and document chunking that's perfect for LLM/RAG applications. Here's why you might want to check it out: Chunklet-py is your friendly neighborhood text splitter that takes all kinds of content and breaks it into smart, context-aware chunks. Instead of dumb character-count splitting, it gives you specialized tools for: Traditional text splitting often breaks meaning - mid-sentence cuts, lost context, language confusion. Chunklet-py keeps your content's structure and meaning intact, making it perfect for: Check out the documentation and GitHub repo for more details! What do you think? Have you worked on similar text processing challenges? Any questions about chunking strategies or the library? Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse CODE_BLOCK: from chunklet import PlainTextChunker chunker = PlainTextChunker() chunks = chunker.chunk( "Your long text here...", max_tokens=1000, max_sentences=10 ) for chunk in chunks: print(f"Content: {chunk.content[:50]}...") print(f"Metadata: {chunk.metadata}") Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: from chunklet import PlainTextChunker chunker = PlainTextChunker() chunks = chunker.chunk( "Your long text here...", max_tokens=1000, max_sentences=10 ) for chunk in chunks: print(f"Content: {chunk.content[:50]}...") print(f"Metadata: {chunk.metadata}") CODE_BLOCK: from chunklet import PlainTextChunker chunker = PlainTextChunker() chunks = chunker.chunk( "Your long text here...", max_tokens=1000, max_sentences=10 ) for chunk in chunks: print(f"Content: {chunk.content[:50]}...") print(f"Metadata: {chunk.metadata}") COMMAND_BLOCK: pip install chunklet-py # For full features: pip install "chunklet-py[all]" Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: pip install chunklet-py # For full features: pip install "chunklet-py[all]" COMMAND_BLOCK: pip install chunklet-py # For full features: pip install "chunklet-py[all]" - Sentence Splitter - Multilingual text splitting (50+ languages!) - Plain Text Chunker - Basic text chunking with constraints - Document Chunker - Processes PDFs, DOCX, EPUB, ODT, CSV, Excel, and more - Code Chunker - Language-agnostic code splitting that preserves structure - Chunk Visualizer - Interactive web interface for real-time chunk exploration - Blazingly Fast: Parallel processing for large document batches - Featherlight Footprint: Lightweight and memory-efficient - Rich Metadata: Context-aware metadata for advanced RAG applications - Multilingual Mastery: 50+ languages with intelligent detection - Triple Interface: CLI, library, or web interface - Infinitely Customizable: Pluggable token counters, custom splitters, processors - Preparing data for LLMs - Building RAG systems - AI search applications - Document processing pipelines - 50+ languages supported - 10+ document formats processed - MIT licensed - free and open source - Active development with comprehensive testing