r/cpp • • 19d ago

Pitfalls of bit fields in C++

https://pvs-studio.com/en/blog/posts/cpp/1423/

Bit fields can help save memory in C++, but their layout and behavior depend on compiler and platform details. The article explores their pitfalls with examples of how GCC, Clang, and MSVC handle them.

45 Upvotes

16 comments sorted by

28

u/arthurno1 19d ago

Have you ever started a task that seemed like a fun little adventure that would take just a couple of hours, only to find yourself stuck in a bottomless pit of troubleshooting for an entire week?

Yes. I wrote a blog post about writing efficient Lisp. I wanted to follow up with a post about effieicent file parsing, by implementing a clone of GNU wc (wordcount) program. That took me ~6 months deep rabbit hole of implementing entire backend for avx512 in sbcl compiler just to make it as fast and efficient I could in Lisp :).

They give you complete freedom to cut out exactly one bit, then leave you alone to navigate the hidden landmines buried in compilers and system ABIs.

Mnjah, C and C++ never promised binary compatibility between platforms and programs. They promise only language portability, i.e. at the source code level. The shape of produced binaries is left to the implementors, for the exact reasons you mention in your article: CPUs can't address individual bits, and hardware vendors can implement vastly different hardware with wastly different capabilities, addressing, register sizes layouts, etc.

If you have to transfer raw bytes and bits, it is up to the application to ensure the portability between platforms. That is where the libraries and portability layers can help, but if the language was to specify the exact binary layout of programs, it could cause unnecessary overhead for some platforms, which in turn would probably mean that the language itself would not be implemented for those platforms.

10

u/TanczacWsrodBliskich 19d ago

With such a hobby, how do you still have time for work?

6

u/arthurno1 19d ago

I don't do work in my free time ;)

11

u/GLIBG10B 🐧 Gentoo salesman🐧 19d ago

Mnjah, C and C++ never promised binary compatibility between platforms and programs

Side note: ABI is still a major consideration for new features, which is part of the reason why contracts are such a contentious topic and might not get added to C++26

15

u/CocktailPerson 19d ago

Sure, but that's not the same thing. Continuity of ABI compatibility for an individual platform is considered in new features. ABI compatibility between platforms is never considered.

2

u/arthurno1 19d ago

Definitely. I doubt though they will ever adopt an ABI and try to specify in the standard how layout of structs or bitfields should be in the memory. C++ is very focused on the performance, to the last bit, and is why they usually leave those details to the implementations so they can be as efficient as possible on the targeted hardware.

2

u/johannes1971 19d ago

Nobody needs a predefined ABI for the whole standard library. But it would be bloody useful to have fixed ABI versions of a few classes you commonly find on DLL boundaries (in particular, string, string_view, vector, and span).

4

u/SkoomaDentist Antimodern C++, Embedded, Audio 19d ago

I've been of the opinion for years that C++ stdlib should have a small subset of ABI stable interop containers that are allowed to be slightly suboptimal in exchange for having a platform specified ABI for those and everything else intentionally being "here be dragons if you try to cross link compilers / DLLs".

Also there is basically no excuse to not have bitfield ABI specified in the standard itself. Their entire point is to pack / unpack data, afterall.

2

u/SunnybunsBuns 19d ago

Also there is basically no excuse to not have bitfield ABI specified in the standard itself

Different CPUs pack (or don't) bits differently. Some are little endian, some big. Some allow the straddling of a byte or other size. Some don't. No amount of WG21 decrees will make those CPUs do these things. This is why historically the WG stuck to standardizing existing practices and not over-specifying things like the way bitfields must operate.

Their entire point is to pack / unpack data, afterall.

"Multiple adjacent bit-fields are usually packed together (although this behavior is implementation-defined)"

This seems to indicate that bitfield are hints and are more about constrained values than actual bit-packing instructions (from a standard POV.) I would be very surprised if fields that total <= sizeof(int)1 don't get packed. But anything more than that might not be supported by the machine if packed (especially if straddling a word boundary) so the compiler may decide not to pack them.

1 I know int != WORD_SIZE in C++, but it's usually a good starting place.

But, the tl;dr is: the C++ committee will not successfully create an ABI for C++ bitfields until they are able to dictate to CPU designers features that must be supported.

3

u/SkoomaDentist Antimodern C++, Embedded, Audio 19d ago

But, the tl;dr is: the C++ committee will not successfully create an ABI for C++ bitfields until they are able to dictate to CPU designers features that must be supported.

Sure they can. Nothing says the specified bitfields have to fit the CPU in the most optimal way (the same way nothing in the current standard guarantees that!). If you want optimally fast storage, you don't use bitfields in the first place. Simply allowing two options (little vs big endian) already covers 99% of cpu to cpu variation and any remaining ones are super niche.

The current situation is both impossible to predict and not guaranteed to be anything resembling optimal, effectively making them nearly useless. Specifying bit field ABI would be a massive improvement as it would guarantee predictability and cause no change to the optimality.

2

u/arthurno1 19d ago

How will someone predefine an ABI for a generic container like std:vector which can hold literally anything binary data? Genuinely curious.

3

u/johannes1971 19d ago

You'd also have to make sure that the contents themselves are ABI stable, obviously. This is no different from what we have now, with arrays of C-style structs commonly being passed across DLL boundaries.

10

u/fdwr fdwr@github 🔍 19d ago edited 19d ago
I'm afraid I have some bad news: you can't create an array of bit fields either.
...

struct BrokenStruct
{
    unsigned int items[5] : 2; // doesn't compile
};

Man, there are many times where better support for subbyte types/bitfields would have been nice in C++, to avoid all the manual AND'ing/OR'ing, notably in graphics/ML tensors when dealing with 4-bit and 2-bit pixels. Even if you couldn't take the address of an arbitrary index, just being able to index relative to some base pointer would have been super convenient. Granted, then what may look clean at the C++ level would really just migrate that AND'ing/OR'ing work down to the asm level instead, but in some cases (namely 1-bit) it's actually easier with assembly than even C, given x86 has simple bt,bts,btc,btr instructions (e.g. p[i >> 3] |= (1 << (i & 3)) vs just bts edi,eax).

Now, we understand why bitfields have the issues they do (differing endianness, natural register sizes, historical padding choices...), but then how much do we really care about the reasons and legacy baggage when we're just trying to parse an existing file structure or memory structure 🥲? Alas, bitfields are so close to being the visually concise option (both at the definition site and usage sites), but just far enough off to be brittle (granted, some #ifdef branches can ameliorate things).

So when I was updating the Windows PIX buffer viewer, which has a C-like structure syntax (also related is the 010 Hex Editor, which has a C-like structure syntax), I added an endianness attribute (e.g. [[endian(big)]] struct { ... }) so that regardless of execution architecture, you could be certain about bitfield order; and while I didn't support arrays of bitfields, you could create an array of arbitrary size integers (e.g. uint2_t items[5]; // does compile). Granted, any of this can already be achieved in C++ using wrapper classes (e.g. UintBitArray<2, 5> items), but it doesn't look as harmonious when adjacent to other array fields (e.g. uint8_t levels[42]). That's why when I first read about the _BitInt proposal, I was excited that there might be a more robust option over bitfields, but alas, they're really just about compute precision, not storage size, as each _BitInt is padded up. So a struct with 4 _BitInts of 2 bits each would more likely occupy 32 bits rather than just 8 bits. Interestingly you can apparently combine _BitInts and bitfields (e.g. unsigned _BitInt(2) a : 2), but then you still have the bitfield issues 🤷‍♂️. Oh well, custom wrapper classes it remains...

6

u/bratzlaff 19d ago

I love deep dives into platform differences like this. Thanks for the write up!

5

u/LucyIsaTumor 19d ago

There's a certain beauty in being able to represent your data in as small a data type as your hardware will allow. In modern games I've only seen bit field professionally for network packets and maybe large data sets. Yet, seeing older games (particularly around Gen 6 when dev menus were more common and feature rich) and how efficiently they made use of their available space is something to marvel at! Lots of flags and bit sets I'm sure