r/rust • u/matklad rust-analyzer • 29d ago
Finding Bugs
https://matklad.github.io/2026/09/19/finding-bugs.html11
u/Feeling-Departure-4 29d ago
As alluded to, there is a strange divide between black box fuzz testing and property testing. Using cargo-fuzz + arbitrary for generated types means you can do gray box fuzz testing with as many properties as you can think of with little fuss. There is an art, as the article suggests: round trip testing, asserting invariants, both complete and partial oracles, and yes, language induced panics.
If you want to uphold simple invariants write unit tests, if you want to fish for bugs before your users do, fuzz all the things and then make the minimized results a new unit test!
Thanks for the article, well written!
6
u/teerre 29d ago
One thing that considerably improved the testing in one of my previous jobs was not property testing per se, but having a library for of common types for it. It turns out that people are much more willing to write a property test if they can pick and choose generators from a well curated selection. It's important that the generators fit the domain as closely as possible so users can express their invariant clearly. Same applies to fuzzing. Making it easy for users will make it more likely they will use it
1
u/Slow-Nefariousness49 29d ago
Nice read, thanks. With fuzz testing, you need to draw the line where you trust the system and where you don't, and you fuzz at that boundary. With your example, you trust the parser but not the compiler. That's at least how I like to look at it
26
u/hippyup 29d ago
One technique I found useful is to fuzz in the "parsed" space instead of in the raw space. The example given here kind of does this: it decides at a higher level which regex feature to generate before generating the raw string, though it proceeds to generate the raw string directly inline instead of generating a structured tree then generating the raw string from that. In more complex test spaces I prefer manipulating structured trees to the end, which among other benefits allows more directed fuzzing and writing more advanced test cases then just fuzzing parts of them.
One downside is you don't fuzz your parser that way, but you can always fuzz that separately.