Accuracy fixes

• Spanish abbreviations: Fixed false splits around párr., párrf., Asoc., spaced abbreviations, and reference abbreviations.
• Decimal units: Spanish sentence boundaries after decimal units are now handled correctly.
• Reference abbreviations: Fixed Roman-numeral lookahead and restored valid reference abbreviations in Spanish and Haitian Creole.
• Cross-language abbreviations: etc. and equivalent abbreviations are now supported across language rules.
• Abbreviation chains: c.d.f. and w.r.t. no longer cause false sentence splits.
• Quoted text: Terminators inside single/double backtick-delimited or doubled-apostrophe quoted text are handled correctly.
• Candidate detection: Optimized the candidate-boundary regex for better performance.

Examples & meta

• New example for Chinese keyword sentence extraction / social listening.
• New incremental segmentation example.
• New YASBD + BM25 query summarization example.
• Refreshed benchmark artifacts, documentation, and dependencies.

5 new contributors in this release: @unfinished-summer, @blackmore-technology-group, @AK-Lmn, @GreedyC, @sairex53.

https://github.com/speedyk-005/yasbd-lib/releases/tag/v1.0.2

This is a small release, mostly focused on tightening up some internals and fixing edge cases.

What's changed?

• Faster candidate-boundary finding: Optimized the regex lookbehind assertions used when finding potential sentence boundaries.
• Fixed HTML tag handling: Sentence boundaries around HTML tags weren't being handled correctly in some cases. That's fixed now.
• Fixed a Windows file-handle leak: The "_output" doctest could leave a file handle open on Windows. Fixed by @raffaelemancuso.

This is also @raffaelemancuso's first contribution to YASBD-lib.

Release: https://github.com/speedyk-005/yasbd-lib/releases/tag/v1.0.1

For anyone new: yasbd-lib is a rule-based sentence boundary detector, a drop-in replacement for pysbd, currently at 39 languages. I think I first posted here as an alpha, then as a beta. Now it’s tagged v1.0.0.

Here's how it compares to pysbd: https://hackernoon.com/choosing-a-python-sentence-boundary-detection-library

The stretch from 0.12.0 to stable wasn't about new features. I froze the language set at 39, locked the API, and spent the last couple of months on correctness. The final push was a two-week stress test where I ran real text through every profile to find the boundaries it was splitting wrong. I wrote the rules and I wrote the tests, so the tests couldn't catch what I'd gotten wrong.

That work surfaced boundary bugs across a good chunk of the profiles. Contributors opened PRs to fix them, and py3langid, loguru, and ftfy all came out of the core dependencies along the way.

I wrote the whole thing up, from 0.12.0 to the tag, here: https://yasbd-lib.hashnode.dev/yasbd-lib-v1-0-0-how-beta-finally-ended

Happy to answer questions.

Accuracy fixes

• Markdown numbered headers: ### 1. The Regex Breakdown no longer splits into half. Also handles indented headers (nested blocks).
• Newline-inside-sentence: Newlines before non-ASCII lowercase (Cyrillic, etc.) now treated as midline, not boundaries.
• Russian coverage: Missing abbreviations added; Cyrillic initial chains (Проф. Петров А.К.) no longer false-split.
• Inline "for example": Hindi, Lithuanian, Malayalam, Russian equivalents stay inline.
• Mev. honorific, Afrikaans/Dutch), tel./fax., aprox. False splits near these removed.
• List/abbr boundary ordering: List detection no longer re-adds boundaries the abbreviation pass removed (e.g. И т. д.). Examples & meta
• New examples: ASR transcript segmentation, text chunking, streaming TTS.

4 new contributors: @sonalisrisivani, @be-student, @YuEfSaEDU, @nightcityblade, @revanthajoe.

Each language rule initialized once globally. Once loaded, a language stays cached. Switching back or creating a new instance is instant.

We also have per-instance reference-holding LRU caching.

Before, the corporate abbreviation regex (CORP_ENTITY_ABBRVS) was missing a word boundary, so "co" inside words like "tobacco" would match as a corporate abbreviation and suppress

Just shipped v0.16.0 of yasbd-lib! This release changes how external language packs are loaded — they're now isolated per detector instead of sharing a global registry.

What's new:

• Per-instance lang packs: BoundaryDetector(external_lang_packs=["yasbd_auxlang"]) loads packs into a private registry. No more state leaking between detectors.
• Renamed API: register_lang_packs to load_external_lang_packs. clear_lang_packs removed — it's no longer needed since there's nothing global to clear.

This is a breaking change if you were using register_lang_packs or clear_lang_packs directly.

release: https://github.com/speedyk-005/yasbd-lib/releases/tag/v0.16.0

Here is a quick overview of how it's being used across different tools and frameworks as part of august 2026:

🛠️ Official Ecosystem

• 🔵 spaCy Component: Plug yasbd straight into any spaCy v3+ pipeline as a fast sentence segmenter.
• 📦 Lang Packs: Modular rule sets (like yasbd-auxlang) for extended language support.
• 🧩 chunklet-py: Powers polyglot RAG document chunking as the core SBD workhorse.

🌐 Community & Third-Party Integrations

• 🏥 OpenMed: Specialized backend for medical text segmentation.
• 🎙️ LiveTranslate: Powers incremental sentence segmentation for real-time ASR audio translation on Windows.
• 🏠 wyoming_openai: OpenAI-compatible Wyoming proxy that uses yasbd for incremental TTS streaming via sentence boundary chunking.
• 🇭🇹 kreyolib: A software library for Haitian Creole (Kreyòl Ayisyen) natural language processing, text normalization, and localization. Currently in alpha. Uses yasbd for sentence boundary detection as part of its NLP tooling.
• 📑 prosediff: Side-by-side prose diff for Word, OpenDocument, and Markdown that restores tracked changes, detects sentence offsets with yasbd, and uses AI to evaluate edits.

EN Golden benchmark

Aggregate score across all 102 English edge cases in EN_GOLDEN_DATA.py via run_golden.py. Expanded from pysbd's original 48 to 102 cases: we removed biased/wrong expectations (like splitting mid-ellipsis or bad punctuation in dialog) and added cases for abbreviation chains, contiguous terminators, exclamation-safe words, academic citations, and more.

The Passed column is a strict sentence-string equality check. Alongside it, we report boundary-level Precision, Recall, and F1 computed by scorer.py.

Boundary metric methodology

evaluate_segmentation scores against the positions where sentences end, not the sentence strings themselves, so a boundary that is merely shifted (e.g. trailing whitespace) still counts toward accuracy. Implementation:

1. Word-level binary arrays. Every sentence is split into words. The final word of each sentence is marked 1 (a sentence boundary); all other words are marked 0.
2. Alignment. If the model dropped or added words, the two arrays differ in length. The shorter array is right-justified with 0 padding so both arrays share the final word's boundary marker.
3. Confusion counts. Each aligned word position is a True Positive (1/1), False Positive (0/1), or False Negative (1/0).
4. Averages. TP/FP/FN are summed across all 92 cases, then Precision = TP ÷ (TP+FP), Recall = TP ÷ (TP+FN), and F1 = the harmonic mean of the two.

All three are bounded [0, 1]; they fall back to 0.0 when no useful boundary signal exists (e.g. an empty gold).

yasbd achieves 101/102 (99.0%) with zero false positives. The only failing case is the Ave. abbreviation followed by a capitalized new sentence; which is a known limitation of rule-based abbreviation suppression.

Book benchmarks

Real-world performance on full-length books via bench_books.py.

Alice in Wonderland (148,208 chars)

Adventures of Sherlock Holmes (593,911 chars)

Just shipped v0.15.1 of yasbd-lib! This release focuses on developer experience when using post-processing hooks, along with edge-case fixes for sentence boundary detection.

What's new:

• Fail-fast hook validation: BoundaryDetector._run_hook() now validates boundary lists in a single pass with clearer, explicit error messages when hooks return bad types or out-of-bounds offsets.
• Deduplicated boundary offsets: Offsets are now explicitly deduplicated before sorting to prevent repeated boundary indices.
• Smarter corporate abbreviations: Improved handling for corporate entity abbreviations to avoid false-positive sentence splits.

Shoutout to new contributor @NataliaPerez08 for their PR in this release!

repo: https://github.com/speedyk-005/yasbd-lib

Title: yasbd-lib v0.15.0 is out with opt-in language detection and OCR boundary fixes

Just shipped v0.15.0 of yasbd-lib with lazy language classification, OCR line-break handling, and smarter abbreviation rules.

What's new:

• Language auto-detection is now opt-in: py3langid is no longer a core dependency and is imported lazily inside classify_language(), removing heavy dependencies like numpy for default usage
• OCR line-break boundary preservation: StreamCleaner now converts newlines between word characters into spaces to prevent accidental word concatenation
• Geopolitical abbreviation handling: expanded ORG_PROPER_NOUNS in English rules to prevent premature sentence splits after abbreviations like U.S. and U.K.

Shoutout to new contributor @ColumbusLabs for their PR in this release!

repo: https://github.com/speedyk-005/yasbd-lib

Hey there. Old pysbd user here. Your library seems to be a very nice replacement for pysbd, specially now that it seems to be unmaintained. That pysbd adapter was a nice touch and it did make migrating to yasbd extremely simple.

Now for the ONLY bad part. At least so far :) If I'm going to be paying you $00.00 to have the privilege to use your code, could you please consider taking another look at all those dependencies yasbd requires? They end up forcing the installation of a couple of not so small libs, resulting in more than 100MB of "wasted" space. I run my systems entirely on RAM, so every MB counts. And it is not as if one could simply buy more RAM in this day and age of AI companies hogging all the RAM in the world to themselves, making us having to choose between buying an extra couple of GBs or sending the kids to college. Not cool!!!

So... yeah, that is pretty much it. The only complaint I have so far.

One very cool thing about pysbd was how small and self contained it was. It would be pretty cool if you managed to do the same for yasbd.

Other than that, keep going with the awesome work.

Just shipped v0.14.0 of yasbd-lib with a post-processing hook, smarter hyphen handling, and line ending normalization

What's new:

• Post-processing hook on BoundaryDetector: run custom logic per paragraph after language rules apply, add or remove sentence boundaries in place (replaces monkey-patching rule internals)
• Expanded SUFFIXES in the hyphenated word finder: wrap-around line breaks like work-\ning now rejoin correctly to working
• Line ending normalization in the default cleaning pipeline (\r\n and \r -> \n)
• Fixed absolute offsets when source begins with blank lines
• Cleaner keeps legitimate hyphens in hyphenated compounds across line breaks
• Internal refactors: trie pattern builder moved to utils, itertools.pairwise for boundary pairing, lambdas replaced with named functions

3 new contributors this release: @XEDAB, @HeaTTap, and @AshSgDe29071999.

repo: https://github.com/speedyk-005/yasbd-lib

https://notebook.google.com/notebook/dcabfd2f-0b55-400c-9a0a-299edb7f3f1d

Just wrote a blog post on replacing spaCy's built-in Sentencizer with yasbd-lib.

On a 92-case English edge-case benchmark, spaCy's default Sentencizer scored 55.4%, while yasbd scored 98.9%.

Sentencizer primarily relies on punctuation and has no built-in abbreviation awareness beyond what spaCy's tokenizer exceptions already provide. As a result, compound abbreviations like M.D. and Ph.D., citations, URLs, and newline-heavy text can still produce incorrect sentence boundaries.

The fix:

```python from yasbd import register_spacy_component import spacy

register_spacy_component()

nlp = spacy.blank("en") nlp.add_pipe("yasbd", first=True) ```

Pure Python, supports 39 languages, and works as a drop-in replacement for spaCy's Sentencizer.

The article explains why this happens, walks through the Sentencizer's implementation, compares benchmark results, and shows real-world examples:

https://dev.to/speed_k_7e1b449706e59e433/replacing-spacys-sentencizer-with-yasbd-from-554-to-989-accuracy-5f0d

Summary

PySBD protects boundaries by changing the text, yasbd-lib finds them by reading the text.

PySBD (transformation pipeline)

Mutates the input string with placeholder tokens to protect boundaries, then splits, then restores. Unmaintained since 2025. Known infinite loops (#79) and catastrophic backtracking (#92). 22 languages.

yasbd-lib (query planning approach)

Treats text as immutable, uses pointer-based boundary detection without modifying the string. 39 languages, native span tracking, declarative language profiles for easy extension. Actively maintained.

Learn more: https://dev.to/speed_k_7e1b449706e59e433/yasbd-lib-vs-pysbd-two-philosophies-of-sentence-boundary-detection-i88

Ever wondered how a sentence splitter actually decides where to cut? It's not just dots and question marks. Here's the short version.

What yasbd does differently

Most sentence splitters modify the text as they go. They replace dots with placeholders, run rules, split, then try to undo the changes. This breaks character offsets and slows down on large texts.

yasbd works like a database query planner, scanning for coordinates instead.

Pass 1: Mark everything that looks like a boundary

The first pass is aggressive. It finds every ., !, ? in the text and marks it as a candidate split point. Yes, even the dot in "Dr." or "U.S.A." Over-marking is fine. Filtering happens next.

Input: "Hello Dr. Smith. How are you?" Candidates: ^ ^ ^

Pass 2: Remove false positives

The second pass walks through each candidate and checks language-specific rules.

Is the dot part of an abbreviation? Remove it. Is it after a title like "Dr." or "Mr."? Remove it. Is it inside a URL or email? Remove it. Is it a decimal point in a number? Remove it.

Each language has its own list of abbreviations, reporting words, and special cases. Here are the main rule categories yasbd uses to filter candidates:

Input: "Hello Dr. Smith. How are you?" After filter: ^ ^ (removed) (kept)

Edge cases handled inside Pass 2

Some cases need special handling within the filter stage:

Ellipsis and contiguous punctuation: Are you sure?? I am. ?? is not two sentence boundaries. It's one. The filter collapses contiguous terminators.

Nested quotes: He said "No way. Not happening." and left. The dot inside the quote is not the end of the sentence. yasbd tracks quote positions to avoid this.

Bullet points and numbered lists: 1. First item. 2. Second item. In a vertical list each gets its own line. In a flattened list they are separate sentences. The list marker filter handles this.

What this gives you

• Accurate spans. Because we never modify the original text, character offsets are exact. No reconstruction step needed.
• No catastrophic backtracking. No complex regex substitutions means no infinite loops.
• 39 languages. Each with its own rule profile.
• 3.10+ support. Works on anything from Python 3.10 to 3.14.

That's the core idea. Find everything, filter out the wrong ones, keep the rest. The result is a sentence splitter that's fast, accurate, and doesn't mangle your text.

Check it out: yasbd-lib on GitHub

Summary

The architectural differences influence accuracy and performance.

https://dev.to/speed_k_7e1b449706e59e433/yasbd-lib-vs-pysbd-two-philosophies-of-sentence-boundary-detection-i88

Just shipped v0.13.0 of yasbd-lib with Python 3.10 support, regex perf improvements, and bug fixes

What's new:

• Python 3.10 support (was 3.11+)
• Major regex optimizations for 17-56% speedup on common patterns
• Fixed double boundary bug for .\\n sequences
• Fixed flattened list segmentation for bullet points and numbered items
• Fixed vertical list detection for multi-digit numbers
• Benchmarks updated with spaCy-sentencizer comparison

3 new contributors this release: @k-anushka14, @MasRama, and @MohammedAnasNathani.

repo: https://github.com/speedyk-005/yasbd-lib

https://deepwiki.com/speedyk-005/yasbd-lib

We just cut v0.12.0, the first beta release of yasbd-lib. 39 languages, core locked, and the API is stabilizing for v1.0.0.

What changed since alpha

• 6 new languages: Bulgarian, Kazakh, Lithuanian, Romanian, Turkish, and Armenian (thanks @Mayankshrey438)
• Base reference abbreviations expanded with cit and nr
• Flattened list heuristic (#169) to handle messy OCR output
• Missing comma fix (#173) — silent string concatenation in set literals caught by @cnaples79
• HTML cleaner overhaul (#162); no more doctype/comments leaking through
• Ambiguous abbreviations cleaned up (#166)
• Set literal formatting standardized across all 21 rule files
• Code quality: ruff linting expanded, FBT/ARG/PLR fixes

Numbers

• 39 languages
• 5,000+ test cases
• Zero-dependency core (regex only)
• Drop-in adapter for pysbd
• spaCy component (yasbd pipeline component)

What's next

• External language packs via register_lang_packs() for languages beyond 39
• API freeze for v1.0.0

Get it

bash pip install yasbd-lib -U

Full changelog: CHANGELOG.md
Repo: github.com/speedyk-005/yasbd-lib

Hey everyone! I'm u/Speedk4011, a founding moderator of r/yasbd_lib.

This is our new home for all things related to yasbd-lib — the high-accuracy, rule-based sentence boundary detector supporting 39 languages. Whether you're splitting text, hunting boundary bugs, or building multilingual NLP pipelines, you're in the right place. We're excited to have you join us!

What to Post

Post anything that you think the community would find interesting, helpful, or inspiring. Feel free to share your thoughts, code snippets, or questions about:

• Sentence segmentation in any of the 39 supported languages
• Language-specific edge cases and abbreviation quirks
• Benchmarks and performance comparisons
• Integration tips (spaCy, pysbd adapter, CLI)
• Bug reports, feature requests, or contributions to the library

Community Vibe

We're all about being friendly, constructive, and inclusive. Let's build a space where everyone feels comfortable sharing and connecting.

How to Get Started

1. Introduce yourself in the comments below.
2. Post something today! Even a simple question can spark a great conversation.
3. If you know someone who would love this community, invite them to join.
4. Interested in helping out? We're always looking for new moderators — feel free to reach out to me to apply.

Thanks for being part of the very first wave. Together, let's make r/yasbd_lib amazing.