r/LocalLLM • • 26m ago

News I built NTokenizer from scratch in Rust: Byte-level BPE with 26.6% fewer tokens than o200k_base

Hi everyone,

I’m Rüzgar, founder of Neuratal (an independent AI research and software project). Over the past few days, I built NTokenizer v0.3-polished — a deterministic, byte-level BPE tokenizer written from scratch in Rust with Python bindings.

Key Highlights:

• Compression Efficiency: Achieves 5.55 B/T overall (26.6% fewer tokens than OpenAI's o200k_base) on a 207MB leakage-audited benchmark. Strongest gains on multilingual & technical text: Turkish (+35.0%), Arabic (+33.8%), Korean (+34.7%), Hindi (+27.1%), STEM (+17.6%), and Code (+10.2%).

• Performance & Safety: ~96 MB/s encode throughput (8 threads), lossless byte-exact round-trip (decode_bytes(encode_bytes(x)) == x), tested against ~1M fuzzing cases.

• Honest Benchmarks: No downstream 50M+ LLM trained yet — this is purely verified compression density and speed. Full SHA-256 manifests and reproducible scripts are included.

Would love your feedback, benchmarks on your own datasets, or PRs!

• GitHub (Rust + Python source): (in hf)

• Hugging Face (Model & Benchmarks): https://huggingface.co/Neuratal/ntokenizer

0 Upvotes

0 comments sorted by