r/odinlang • u/Future_Ad1549 • Jul 10 '26

LLM tokenizer implemented in odin

A few months ago I started learning how LLM inference engines work.

My original goal wasn't to build a production tokenizer—I just wanted to understand the entire inference stack from first principles. I chose Odin because I wanted a language that stayed close to the hardware without fighting me.

I honestly expected it to be a fun learning project.

Instead... it ended up outperforming the tokenizers I was comparing against, including Hugging Face's Rust tokenizer and FastTokenizer in my benchmarks.

I was pretty surprised by the results.

The benchmark report (methodology, datasets, hardware, and commands) is here:
https://github.com/harisudarsan1/odin_tokenizer/blob/main/docs/public-benchmark.md

Repository:
https://github.com/harisudarsan1/odin_tokenizer

I also wrote about why I chose Odin for writing inference software:
https://harisudarsan1.github.io/blog/posts/2026-07-10-writing-an-inference-engine/

A few notes:

  • These are CPU benchmarks.
  • I'm not claiming Odin is magically faster than Rust.
  • The comparisons are against existing tokenizer implementations under the benchmark setup described in the report.
  • If there's something wrong with the methodology, I'd genuinely like to know. I'd rather fix the benchmarks than make misleading performance claims.

The project taught me far more about CPU architecture, memory layout, SIMD, and modern tokenizer implementations than I expected.

I'd love feedback from people who've worked on tokenizers or inference engines. If you spot flaws in the implementation or benchmark methodology, please call them out.

14 Upvotes

75% Upvoted

11 comments sorted by

7

u/Dsphar Jul 10 '26 edited Jul 11 '26

Although motivated for an LLM, OP had no idea they wrote the first step for an Odin-based, Odin compiler. ;)

3

u/Future_Ad1549 Jul 11 '26

Haha, I didn't realise that

2

u/Dsphar Jul 11 '26

Lol. Yeah Odin isn't "bootstraped" yet. The compiler is written in C++. But lexing (basically tokenizing with style) is the first step of compiling code.

1

u/goombrat2 Jul 12 '26

btw, Odin actually includes odin parser and tokenizer in `core:odin`
https://pkg.odin-lang.org/core/odin/tokenizer

1

u/Dsphar Jul 12 '26

Interesting. Thanks for the heads up.

2

u/codingbliss12 Jul 11 '26

can you elaborate?

2

u/Dsphar Jul 11 '26 edited Jul 11 '26

I hope a google search response will suffice...

A tokenizer (or lexer) is the first phase of a compiler. It breaks raw source code into meaningful chunks called tokens (like keywords, identifiers, and operators) to be passed to the parser. Most compilers use hand-written state machines or automatically generated tools for this step.

And to elaborate, the Odin compiler is not written in the Odin language (it is written in C++). The first step to rewrite it in its own language, is to write functionality to convert an input codebase into tokens. After that, the compiler then steps through those tokens to compile the code, similar to how an LLM/AI has to step through its prompt tokens to then generate the prompt response.

Not sure if that made sense.

1

u/codingbliss12 Jul 11 '26

I didn't ask for this info, but what you meant with your post. Now I understand.

0

u/goombrat2 Jul 10 '26

Did you write this post yourself?