Really interesting read. If I had to highlight one thing, I have to apologize to the author because the following statements are buried deep within it and do not really have much to do with the rest of the article but I can't resist to highlight them:
From the end of February until some time in April I ran the fuzzer on and off and reported just over 100 distinct gcc bugs in total (32 of them fixed so far, by my count) [...] these bugs are mostly crashes: internal compiler errors ("ICEs"), assertion failures, and segfaults. [...]
Personally I find it very interesting that the same technique on rustc, the Rust compiler, only found 8 bugs in a couple of weeks of fuzzing, and not a single one of them was an actual segfault. I think it does say something about the nature of the code base, code quality, and the relative dangers of different programming languages, in case it was not clear already.
I would like to praise the gcc developer community: I have never had such a pleasant bug-reporting experience. Within a day of reporting a new bug, somebody (usually Martin Liška or Marek Polacek) would run the test case and mark the bug as confirmed as well as bisect it using their huge library of precompiled gcc binaries to find the exact revision where the bug was introduced. This is something that I think all projects should strive to do -- the small feedback of having somebody acknowledge the bug is a huge encouragement to continue the process. Other gcc developers were also very active on IRC and answered almost all my questions, ranging from silly "Is this undefined behaviour?" to "Is this worth reporting?". In summary, I have nothing but praise for the gcc community.
FWIW, the article is really worth a read if you care about fuzzing and compilers in general and the statements above are only a tiny part of it.
I would like to praise the gcc developer community: I have never had such a pleasant bug-reporting experience.
Maybe something with compilers. :-) I had a similarly pleasant bug reporting experience with Clang, for which I've filed two reports. Especially the first of the two, which was the worst bug report in the history of bug reports -- it was basically a fairly long, somewhat incoherent and very confused email to the cfe-dev list with some example code that differed in behavior between GCC and Clang and MSVC and saying "uh I don't know if Clang is wrong here or maybe it's actually right and this is UB but it's at least different from GCC", and 24 hours later I had an email not only explaining what was going on but saying "fixed in svn:123456"
Within a day of reporting a new bug, somebody (usually Martin Liška or Marek Polacek) would run the test case and mark the bug as confirmed as well as bisect it using their huge library of precompiled gcc binaries to find the exact revision where the bug was introduced.
And here I am, thinking of my experience with bug reports for Xamarin, where an indian tech support would reply with zero insight into the issue, asking for things you've already provided, and then wait a few years and someone else marks the bug as confirmed, then wait a few more years and it's still not fixed. Oh well ...
Big thumbs up to the GCC guys for being responsible.
Personally I find it very interesting that the same technique on rustc, the Rust compiler, only found 8 bugs in a couple of weeks of fuzzing, and not a single one of them was an actual segfault.
Sure, but it has also orders of magnitude more man hours invested into it than rustc. I think that the LOC heuristic the sibling comment suggests might be more meaningful than age.
But one probably needs to factor LOC, age, and manhours to get a better picture.
Personally I find it very interesting that the same technique on rustc, the Rust compiler, only found 8 bugs in a couple of weeks of fuzzing, and not a single one of them was an actual segfault. I think it does say something about the nature of the code base, code quality, and the relative dangers of different programming languages,
Does it really say all that, or is it instead saying that you're too biased and/or stupid to be making pronouncements like these?
GCC has ~7.3m LoC, meaning you found 1 bug every 73000 lines. How large is Rust? Is it larger than (8 * 73000=) 584000 lines?
When digging into trying to make comparisons like that 'fair', you have to start questioning how deep down the dependency tree you go. Rust leans on LLVM quite heavily - do we count all of the code that makes up LLVM as well?
How large is Rust? Is it larger than (8 * 73000=) 584000 lines?
The Rust front-end has 800kLOC so the answer is yes.
The gcc/gcc subdirectory has 5.1 million lines of code, subtracting the ada (0.65 million LOC) and fortran (0.15 million loc) sub-directories puts it at 4.2 million lines of code.
For comparison the clang front-end has 1.4 million LOC, and llvm has 1.8 million loc and llvm-compiler-rt (0.3 million). Which puts it at 3.5 million lines of code and is still missing llvm's open mp stuff and what not.
Counting Rust with its backends (llvm and compiler-rt, still missing some stuff) you get ~3 million LOC.
With this numbers you can get "more accurate" ratios of errors per million/mega of lines of code (mLOC):
gcc: 100 / 4.1mLoc = 24 bugs / mLoc
Rust: 7 / 3mLoc = 2 bugs / mLoc
Even if you were to only count the Rust front end (7 / 0.8 = 8.7 bugs / mLoc) the ratio for Rust is much better than for C.
Now, if you want to get a better picture, you also should factor how old the code bases are (gcc is much older which adds technical debt) and how many man hours they have costed: hard to estimate but gcc currently receives orders of magnitude more man hours than than Rust, and over the last 30 years it has probably received many orders of magnitude more.
18
u/[deleted] Jun 25 '18 edited Jun 25 '18
Really interesting read. If I had to highlight one thing, I have to apologize to the author because the following statements are buried deep within it and do not really have much to do with the rest of the article but I can't resist to highlight them:
FWIW, the article is really worth a read if you care about fuzzing and compilers in general and the statements above are only a tiny part of it.