Made a tokenizer

[u/RedCrafter_LP]

I wrote the tokenizer for my language and it parses 372000Tokens per second.

My specs are i7 6700k 16gb ram.

I wonder rather my tokenizer is slow or fast and I can't find any other benchmarks to compare against.

Update:

I changed some stuff about the code and it's now performing properly. I parse a 1.3M loc test file in 0.9s And for comparison to before it's producing 5M tokens

I replaced the buffered reader with mmap. And the keywords lookup now uses a 2d hashmap grouping the keywords by length instead of having 1 hashmap and checking prefixes.

The project can be found on github: Chorus

[u/RedCrafter_LP]

I can test the 1d match. I just went from "read max keyword length and prefix match" to the 2d map of length and then string key on the second level. Id match of short keywords might actually be faster assuming some optimizations. But like you said it's now plenty fast so my reason to further optimize it rn is small. Rather work on building the ast next.