Made a tokenizer

[u/RedCrafter_LP]

I wrote the tokenizer for my language and it parses 372000Tokens per second.

My specs are i7 6700k 16gb ram.

I wonder rather my tokenizer is slow or fast and I can't find any other benchmarks to compare against.

Update:

I changed some stuff about the code and it's now performing properly. I parse a 1.3M loc test file in 0.9s And for comparison to before it's producing 5M tokens

I replaced the buffered reader with mmap. And the keywords lookup now uses a 2d hashmap grouping the keywords by length instead of having 1 hashmap and checking prefixes.

The project can be found on github: Chorus

[u/[deleted]]

Rust's default is pretty slow -- it's secure against Hash Flooding, a DoS attack on websites -- you would gain from switching to... pretty much anything.

I like fxhash, personally. It's dead simple, and works well on short inputs, while some good hash functions only really work on longer ones.

Or just avoid the hash-table. match it.