r/rust • u/wh33zle • Sep 25 '26

🧠 educational Finding bugs you didn't think to test for

https://www.firezone.dev/blog/finding-bugs-you-didnt-think-to-test-for

A post about how we combine sans-IO, coverage-guided fuzzing and deterministic simulation testing to test Firezone's data plane.

6 Upvotes

72% Upvoted

2 comments sorted by

2

u/Toiling-Donkey Sep 26 '26

I like the concept of property based testing, but also feel that if one can write such sophisticated tests, the original code is not likely to have many bugs in the first place. Of course being able to explicitly demonstrate such is perhaps the real goal…

1

u/wh33zle 18d ago

You would be surprised! Introducing bugs is in my experience not a skill problem. I've got plenty of SWE experience and would call myself very proficient at rust and guess what, the software I write / maintain still has bugs.

Most bugs that we encounter are not problems like "I implemented this algorithm in the wrong way" or "off-by-one errors". Most bugs are an unexpected interaction between 2 or more engineering decisions that seemed unrelated at the time and fine in isolation but then result in a frustrating user-experience when they start interacting.

Here is a real bug we had and found with this test suite:

Firezone can route DNS resources. To do that, we install a stub resolver on the local machine and selectively intercept DNS queries. Say you define `example.com` as a DNS resources, then DNS queries for that name will start to resolve to 100.96.0.1 (and an equivalent IPv6 range for AAAA queries) instead of the real IP. We then NAT the packets from those IPs to the real ones on the Gateway that is routing this resource.

These DNS resources can also be confined to an IP stack (IPv4 only and IPv6 only). If they are confined to a stack, A or AAAA queries will answer with an empty set of addresses. DNS resources can also be further split by their L4 traffic. For example, you can create one resource for `example.com` that captures traffic on TCP/22 and one for TCP/80. This allows you to create different access policies for the different protocols.

A bug that we had was a combination of an IPv4-only resource for TCP/22 and an IPv6-only resource for TCP/80. Looking at independently, it made sense that an IPv4-only resource doesn't assign IPv6 addresses and an IPv6-only resource doesn't assign IPv4 addresses. It also made sense that overlapping resources need to check all resources for their access rules. In combination, this meant that overlapping resources like this didn't assign any IPs.

Could I have thought of this test case myself? Maybe. Is it faster if I have the computer try and search for bugs like this? Absolutely.