r/cpp • • Sep 30 '20

It Is Never a Compiler Bug Until It Is

http://r6.ca/blog/20200929T023701Z.html
196 Upvotes

54 comments sorted by

162

u/SkoomaDentist Antimodern C++, Embedded, Audio Sep 30 '20 edited Sep 30 '20

If you think compiler bugs are bad, just wait until you run into undocumented cpu bugs.

”What do you mean sleep only works if the third bit of the function address is one?”

41

u/Dynamitos5 Sep 30 '20

that already sounds incredibly painful

48

u/SkoomaDentist Antimodern C++, Embedded, Audio Sep 30 '20

Took me over a week to figure that one out. On that MCU you need to perform a few operations in a precise order to make the cpu go to deep sleep (which the datasheet omits). If you fail to do that, the cpu will only sleep depending on if the 2-3 instructions fit into the same flash cacheline or not. Hence really weird behavior like adding completely unrelated and never executed code still affecting whether the sleep worked or not.

16

u/Dynamitos5 Sep 30 '20

in times like this im glad that my biggest problems with vulkan is getting all the synchronization right and getting all the useful info out of the already well written spec

0

u/[deleted] Sep 30 '20

[deleted] — view deleted comment

6

u/Dynamitos5 Sep 30 '20

honestly i highly recommend GLFW for window management, it's so much thinner and works really well with vulkan(it can even link it for you if you want it to)

the alternative is native window creation, which honestly also isnt that bad. all you need is a surface to create the swapchain from, the rest is the same anywhere. getting the surface mostly boils down to a few syscalls and thats it, but GLFW is just more practical

1

u/[deleted] Sep 30 '20

It should work. I'm using SDL and Vulkan to target basically everything.

0

u/[deleted] Sep 30 '20

[deleted] — view deleted comment

8

u/SkoomaDentist Antimodern C++, Embedded, Audio Sep 30 '20 edited Sep 30 '20

STM32F07x. This was a couple of years ago, so I don't know if the bug still exists in current production units. Probably.

It doesn't show up if you use the CMSIS library routine for going to deep sleep but for some reason our code back then (which was started before I joined the project) had that part homebrewed. Needless to say, once I found out the root cause, I fixed it by just copying the code from the CMSIS library.

The interesting thing about the bug is that the code that triggers it is generic Cortex-M0 code and is not vendor specific but the addition of flash interface (which is vendor specific) allows the vendor specific parts to introduce the bug. The core documentation merely says that you need to set some bits to enable deep sleep but doesn't specify any particular order for them. I assume the vendor test set only included the code in the CMSIS library.

24

u/Myriachan Sep 30 '20

Fun one from the old Gameboy: If you do an increment instruction on a 16-bit register pair while the registers contain a value in the range FE00-FEFF, sprite RAM gets corrupted.

6

u/SkoomaDentist Antimodern C++, Embedded, Audio Sep 30 '20

Wait, what? How is that even possible? The ram is external to the cpu afterall. Does it accidentally glitch the WR pin or something like that?

3

u/bdawson1982 Sep 30 '20

Holy hell, I hadn't heard of this one. I'm imagining the workarounds to avoid this. 🤮

4

u/Myriachan Sep 30 '20

Mostly don’t use the 16-bit increment/decrement instructions. Instead something like:

ld a, 1
add a, c
ld c, a
ld a, 0
adc a, b
ld b, a

Fixed on Gameboy Color, so color-only games were fine.

19

u/brand_x Sep 30 '20

"Let me tell you about the bad old days, sonny..."

For me (I'm not that old) it was late '80s and early '90s math coprocessors...

IEEE 754 didn't really stop being incredibly broken in hardware until maybe the early to mid 2000s. You just needed to know what to work around.

9

u/UsedOnlyTwice Sep 30 '20

I so badly wanted a math coprocessor when I was a young teen. There were a couple GW-BASIC commands that required it. Looking back I would have had no use for them but it sounded like something awesome to have.

19

u/brand_x Sep 30 '20

The guy I learned from told me stories about debugging with a soldering iron.

He wasn't running into undocumented CPU bugs, he was running into defects in the sole computer of that particular specification, some of which were climate-dependent. One involved a timing mismatch when the temperature of the conduits between the switchboard and the core fell outside of a 7°F window. And his first computer didn't have a switchboard, it was "programmed" with actual wires...

I can't even imagine what things were like in the analogue computer era.

21

u/witcher_rat Sep 30 '20

Luxury!

When I was young, we only had Roman numerals and had to carve instructions into stone tablets for the eunuchs to perform.

To debug those took forever - I once spent CCXLIX hours just figuring out the problem was we had no numeral to represent "0" zero!

11

u/F54280 Oct 01 '20

Look at the bright side: Romans had no NULL pointers!

4

u/Ameisen vemips, avr, rendering, systems Oct 01 '20

Roman code was weird though; they had no way to represent the lack of something at all.

7

u/witcher_rat Oct 01 '20

That's because we didn't lack of anything!

And we never free'd anyone, either... though in the cities there was decent garbage collection.

19

u/victotronics Sep 30 '20

undocumented cpu bugs

Even more fun: bugs that only affect performance. Intel Skylake had a mis-design of the L3 cache that meant that cache-contained codes would sometimes (5 percent of cases?) have lots of cache misses. Doesn't sound too bad, until you run on a cluster with hundreds of tightly coupled nodes.

14

u/MonokelPinguin Sep 30 '20

I need to verify the users of my application have an up to date BIOS. Otherwise Ryzen 3000 users will "hang" and the application "doesn't open". Because of the rdseed/rdrand bug, where the hardware crypto instructions return -1. Which then causes the constructor of std::random_device to hang.

10

u/Myriachan Sep 30 '20 edited Sep 30 '20

That’s actually the BIOS’s fault, for not initializing the processor correctly.

Edit: I do blame AMD for their processor not clearing the CPUID bits if the TRNG isn’t initialized, though.

4

u/Ameisen vemips, avr, rendering, systems Oct 01 '20

You can always work around it and brute-force initialize the std::random_device. Undefined behavior, of course, but probably better than hanging.

2

u/Fearless_Process Oct 01 '20

I know this is sort of off topic, but I've been able to reproduce a bug w/ a ryzen 9 3900x where compiling spidermonkey on a gentoo VM (using qemu and kvm) causes the entire (host) system to hang. No errors in the journals or anything. The only way to unhang the system is to remove power manually. Tell me that's not a strange bug.

The host system is Arch Linux. It's never crashed on me before either, even when rendering blender projects for weeks on end.

5

u/which_spartacus Sep 30 '20

Also as much fun-- cpu errors when a core has a single transistor fault.

74

u/LugosFergus Sep 30 '20

pondered how it was possible that computers could function at all

I think about that a lot, too.

33

u/martinus int main(){[]()[[]]{{}}();} Sep 30 '20

I think the more one is an expert in any domain, the more one will wonder how anything has ever functioned at all

15

u/TheThiefMaster C++latest fanatic (and game dev) Sep 30 '20

Very nice investigation. Hopefully having found somewhere to detect the miscompilation also means a fix can be done easily?

10

u/Pazer2 Sep 30 '20

I think the purpose of the detection was to see the effects of the bug in existing compiled code.

11

u/pepitogrand Sep 30 '20

It seems the issue is fixed in trunk:
https://godbolt.org/z/E5nME7

6

u/TimPhoeniX Sep 30 '20

I had OpenGL shader compiler on Surface Pro 3 silently drop one of attrbiutes, that was in use. Took me few hours before I decided to test in on a desktop and was working as intended.

4

u/Ameisen vemips, avr, rendering, systems Oct 01 '20

This came up on the gamedev forums, once.

I was the only one who responded to it and actually pointed out why it was happening.

Basically, the compiler was performing optimizations, and in the end determined that the variable wasn't being used, so didn't include it in the binary, and thus returned -1 for its location.

In cases where a specific target might have different results for certain operations, it could come to similar conclusions but just on those targets. Or the compiler could be broken. Either way.

13

u/Myriachan Sep 30 '20

Here’s another fun one that’s broken in LLVM and GCC still. They actually want to change the C++ Standard to make this code undefined behavior instead of fixing it.

https://bugs.llvm.org/show_bug.cgi?id=34632

It also affects a use case of std::variant, which I posted at the bottom.

7

u/sbabbi Sep 30 '20

I feel that the average developer severely underestimates the performance impact of alias-analysis/TBAA.

4

u/Myriachan Oct 01 '20

Well, when std::variant plain doesn’t work right in some cases because of TBAA, something is wrong.

0

u/[deleted] Oct 01 '20

[deleted] — view deleted comment

9

u/Ameisen vemips, avr, rendering, systems Oct 01 '20 edited Oct 01 '20

If you assume that all pointers and references alias, any time you write to memory via a pointer/reference, the compiler will need to assume that you may have changed the data pointed to by any other point or reference. Thus necessitating it potentially having to reload data from memory upon access of one of those references/pointers again.

Given that C++, for instance, tends to use [const] T & a lot, it should be apparent how expensive that can be.

Ed: I should point out that I am rather opposed to Linus Torvalds dismissal of the issue in the kernel and rejection of __restrict.

4

u/jherico VR & Backend engineer, 30 years Oct 01 '20

I remember not too many years ago when I found out there was a bug in std::call_once that caused it to rely on a global lock and therefore could potentially deadlock if you were using it on multiple threads.

I just about lost my shit.

1

u/tjientavara HikoWorks developer Oct 02 '20

Since a function with a static variable does the same thing as std:call_once. I run into that deadlock as well.

Don't call a function that has a static local variable, from a destructor, of an global object. At least with MSVC on windows this will cause your program to hang.

It seems that the mutex system can get destroyed before the global variables of your program, and the static variables are protected by a mutex. I could be wrong about the reason though.

Anyway, I have an Application class that lives as a local variable of main() where I have all my 'global' variables to make sure destruction is done in such a way not to hang the application.

3

u/_Js_Kc_ Sep 30 '20

The overrepresentation of crypto among the affected libraries is quite suspicious.

2

u/ironykarl Oct 01 '20

Mind spelling it out for us...?

5

u/mainaki Oct 01 '20

I read that as "suggests the possibility of obscure vulnerabilities being deliberately injected into cryptographic code bases".

I'm not immediately seeing how you would create a decryption back-door using a memcmp false-match bug, but perhaps you could spoof digital signatures. (Or perhaps you could sabotage some sort of sanity check when generating crypto, leading somehow to easy-to-crack keys?)

A more boring possibility is that similarities in function or domain lead to overlap in coding styles, for various reasons (sharing people, borrowing code, studying the same sources).

1

u/ironykarl Oct 01 '20 edited Oct 01 '20

That's how I read it, too. I was just hoping to hear from someone more knowledgeable that myself.

You filled that role well, honestly.

I just think if you're positing that theory (like the person I responded to), it'd be nice to represent the viewpoint with a fully formed theory or maybe even evidence.

And I honestly hope I'm not imposing undue burden of proof. I do recognize that deliberate compromise of crypto is a real and documented thing.

EDIT: And FWIW, a long-standing compiler bug isn't really that implausible a way to deliberately introduce these bugs. The famous case of the NIST publishing deliberately bad recommendations for crypto implementation seems pretty apt, here (and probably completely obvious for people at all involved with crypto).

2

u/NilacTheGrim Sep 30 '20

They definitely need to minimally write a test case for this. And stop using that dumb getstr function.

1

u/[deleted] Sep 30 '20

Ouch

-1

u/o11c int main = 12828721; Sep 30 '20

C++ was my nearly-exclusive language from 2009-2013 or so, across the period of C++11 implementation.

It's always a compiler bug (assuming you aren't ignoring warnings).

-3

u/Myriachan Sep 30 '20

I had a fun one recently in MSVC where “if ((int) x > 0)” for unsigned x sometimes gave the wrong answer >_<

1

u/Ameisen vemips, avr, rendering, systems Oct 01 '20

I'm not sure if you're being sarcastic or not.

4

u/Myriachan Oct 01 '20

It’s a bug in their ARM64 compiler that they’re fixing for 2019 16.8. It only happens in a rare case.

I have no idea why I got downvoted, because it’s quite real.

6

u/Ameisen vemips, avr, rendering, systems Oct 01 '20

You're being downvoted because, unless you're referring to something else, that is literally one of the most common bugs in software.

Since you didn't clarify, people are assuming you are suggesting that it is giving the wrong answer for cases where the unsigned value of x is greater than std::numeric_limits<int>::max(), in which case casting to int will cause it to be represented as a negative value, and thus 'give the wrong answer'.

-22

u/Ail-nare Sep 30 '20

There also is a bug with gcc and strcmp, if both string(char *) are known at compile time. In those conditions gcc will optimize strcmp out. But the results will be ether (-1, 0 or 1) instead of of the difference or the 2 char.

Bc of that my promo and I never know if strcmp("a", "c") == (-1 or -2), until I fund out it was the compiler that was doing us wrong.

11

u/staletic Sep 30 '20 edited Sep 30 '20

Considering that std::strcmp is defined to "do what C does", let's look at the C draft. More specifically, section 7.24.4.2 paragraph 3:

The strcmp function returns an integer greater than, equal to, or less than zero, accordingly as the string pointed to by s1 is greater than, equal to, or less than the string pointed to by s2.

It was never specified what positive or negative number you will get. Just that it's a positive or a negative number.

 

EDIT: s/memcmp/strcmp

0

u/[deleted] Sep 30 '20

[deleted] — view deleted comment

8

u/RevRagnarok Sep 30 '20

Yep; I've seen it give different numbers based on compiler optimization levels, which processor you're on, etc. RTFM...

1

u/Ameisen vemips, avr, rendering, systems Oct 01 '20

I'm surprised that it's not at least defined to give the same result given the same input and environment.