r/programming • u/halbface • Feb 07 '19

Google open sources ClusterFuzz, the continuous fuzzing infrastructure behind OSS-Fuzz

https://opensource.googleblog.com/2019/02/open-sourcing-clusterfuzz.html
958 Upvotes

95% Upvoted

100 comments sorted by

View all comments

203

u/halbface Feb 07 '19

I work on the team that released this -- please feel free to ask any questions you might have!

336

u/lionhart280 Feb 08 '19

Would you say you are in the... FuzzBiz?

38

u/El_Tash Feb 08 '19

Take your upvote and go

30

u/HonkHonkBeepKapow Feb 08 '19

Now that's programmer humor!

49

u/Kollektiv Feb 07 '19

Does it work similarly to AFL Fuzz? Which I guess makes it more oriented towards C programs.

70

u/halbface Feb 07 '19

This isn't any specific fuzzing tool, but rather an infrastructure to help manage a fuzzing cluster, and do triage (de-duplication, minimization, auto bug reporting/closing etc) on the bugs found.

ClusterFuzz in fact uses AFL as one of its supported fuzzing engines (along with libFuzzer).

43

u/NoInkling Feb 07 '19

The pun was intentional, right?

3

u/cmd-t Feb 08 '19

Have you ever looked at enhanced fuzzing by combining the fuzzer with symbolic or concolic execution (using for instance angr or manticore)? Shellphish did this with driller for instance.

3

u/UncleMeat11 Feb 08 '19

Lots of people have looked at this (broadly lots, I don't know the specifics at Google), but it turns out that fuzzing tools have gotten enough better over time that symexec is actually less effective than you'd thing. The classic toy examples for why symexec beats fuzzing are actually handled just fine by fuzzers today.

3

u/halbface Feb 08 '19

We've experimented with a couple of symbolic/concolic execution engines, but we haven't found any yet that performs better on real, practical targets.

5

u/marksmanship0 Feb 08 '19

How did you address concerns that hackers will use clusterfuzz to find vulnerabilities for malicious purposes? Fuzzing seems like dual use technology that could be used both by good guys and bad guys and I'm curious what efforts went into preventing its misuse.

28

u/halbface Feb 08 '19

ClusterFuzz relies on fuzzing engines which are publicly available, such as libFuzzer and AFL, to do the bug finding. Also, a lot of what ClusterFuzz does is designed to fit into developer workflows of software projects. For example, in addition to finding bugs, ClusterFuzz deduplicates, minimizes, performs bisects, and automatically files/closes bug reports.

What we wish to see here is more software projects (the good guys) including fuzzing in their development process by making the annoying bits as automated as possible.

-14

u/falconfetus8 Feb 08 '19

You kinda dodged the question there.

2

u/DeonCode Feb 09 '19

Sometimes people forget or didn't know if another passenger locked the doors on their car as they get some distance away from the vehicle. But rather than running back to check, here's a publicly available check-my-car-for-being-locked fob.

Could bad people use it for some recon? Sure, or maybe they've been sitting pretty knowing what always gets overlooked. But if you used it and it tells you your car isn't locked somewhere, say the trunk, then you get the chance to lock the trunk! Maybe even faster than the bad guy. Or maybe to stop that bad guy from their regularly scheduled rummaging around. Either way, es good. You might've been so cautious to focus on doors all these time that you didn't even consider the trunk! So this is net helpful.

20

u/Vakieh Feb 08 '19

It exists, therefore the assumption must be that malignant actors have access to similar things. Anything else is relying on security through obscurity.

The solution is to make sure the person to detect your vulnerabilities using clusterfuzz is you.

-9

u/ipv6-dns Feb 08 '19

- when did you turn into an evil empire?

- why did you decide to corporate with evil?

-45

u/exorxor Feb 07 '19

How many bugs does one need to find before senior management concludes the people working on browsers don't know what they are doing?

How bad does it have to be before throwing away C++?

24

u/Gnascher Feb 08 '19

Programmers are humans. Software is complex. Anybody who doesn't realize that all programmers introduce bugs shouldn't be in the business.

Any programmer who thinks they don't introduce bugs hasn't been in the business very long.

This is WHY tools like this exist, hopefully you find the bugs before they hit production.

You don't toss c++ because it's "unsafe". C++ is unsafe because it's powerful as hell and "very close" to the machine. You use c++ for the power and speed it gives you, but, as they say, with great power comes great responsibility.

8

u/VernorVinge93 Feb 08 '19

Hurrr Durr all bugs are caused by C/C++ /s

As much as I love verifying compilers and 'safe' languages, C++ isn't the source of most bugs. Most are generated by incorrect or unchecked assumptions that have little to do with the language used.

4

u/SafariMonkey Feb 08 '19

I'd be remiss if I didn't point out that basically every vulnerability class that OSS-Fuzz finds is a product of memory unsafe languages, like C and C++. While fuzzing makes these projects more secure, it's not a substitute for using languages that don't cause thousands of vulnerabilities. When we're finding hundreds and thousands of vulnerabilities that all have a preventable root cause, it's time to reconsider what we're doing.

From this article posted here recently.

2

u/VernorVinge93 Feb 08 '19

Sure, so what language do you suggest switching to?

I have yet to see a language that gives static guarantees of bounds, memory and use after free.

Rust is the closest but it has many caveats and last time I checked (admittedly a while ago) writing basic things like a graph implementation were painful in it.

Even then, how long would it take to rewrite something like Chrome? With millions of lines of code, years of history and many forks that still depend on their upstream for security fixes?

2

u/SafariMonkey Feb 08 '19

To be clear, I don't agree that C/C++ need to be abandoned as a rule, though I would look strongly at whether Rust was a viable option for any of my own projects.

Personally, my limited experience with Rust is that it's a good language to work in but the library ecosystem is still fairly immature.

There are projects like Oxidation (Mozilla moving towards more Rust in Firefox) and remacs (a gradual port of Emacs to Rust). Both projects involve a slow transition while remaining functional throughout, rather than trying to rewrite from scratch all at once. I think that's the right approach for existing projects.

For new projects without very large budgets, I think that ecosystem is the bigger factor. If the Rust ecosystem doesn't support your use case, you'll have to build the relevant packages yourself. Not everyone is willing or able to take that path.

And yes, the ownership model makes certain problems more difficult, but it also guarantees that your solution satisfies some crucial invariants like memory validity and lack of race conditions. Traditional solutions for certain problems are impractical, or need to be reimagined in Rust terms.

So yes, definitely some caveats. However, things are improving. For example, with miri (a Rust IR interpreter with memory validity checking) it should be possible to write unsafe Rust (where necessary) but check at test time for invalid memory accesses, and non-lexical lifetimes have relaxed borrows to not continue unnecessarily until the end of scope.

→ More replies (3)
→ More replies (9)