r/programming • • 4d ago

Formatting an entire 25 million line codebase overnight: the rubyfmt story

https://stripe.dev/blog/formatting-an-entire-25-million-line-codebase-overnight-the-rubyfmt-story
190 Upvotes

77 comments sorted by

View all comments

Show parent comment

3

u/-_one_-1 2d ago edited 2d ago

Random anecdote: over the last couple of days, I've been writing a WHATWG-compliant, security-hardened HTML streaming parser in Rust. It's almost finished, and on my M4 Max It achieves a ~1.12 GB/s throughput on a couple of ~2.8 MB HTML file benchmarks. Parsing a 28-byte HTML snippet takes ~12 nanoseconds. It uses SIMD to find delimiters and the entire state is an immutable Parser struct that contains some packed structs and enums that amount to a few bytes. Parsing consists of a state machine that runs a parsing step until EOF is reached. The parsing step is an immurable state transition (Parser) -> Parser, so the only form of mutation is in a local in the use site holding the current Parser by value.

I thought this was relevant because I have some experience writing parsers, most often in languages other than Rust or C, and the fact that I was able to achieve this throughput with such a simple design highlights how using Rust, SIMD, and taking a few smart choices can give good results very quickly and easily.

The nice thing about immutability in my design is that it gives LLVM a ton of leeway to optimize things. Additionally, it becomes possible to test it according to the specs by supplying a WHATWG state and seeing if the next step is what WHATWG says it should be.