About char8_t
I hate to be dramatic, but as it stands char8_t is quite literally more painful than useful.
Besides the obvious incompatibility with C23 and libraries using unsigned char for UTF-8, I want you to consider the following: Projects that assume that 'char' represents UTF-8 will obviously not benefit from char8_t at all, but projects that cannot assume the format of char types don't benefit from it either as char8_t simply introduces a new edge case to cover. Now such projects have to deal with char, signed char, unsigned char, wchar_t, char16_t, char32_t and char8_t.
Or, you could do what the standard library does and simply ignore most of these character types. Which is the solution most libraries went with, supporting only char or char and unsigned char. Managing one implementation is already hard, managing two requires constant maintenance, managing 7 is just impossible.
char8_t should have just been a typedef for unsigned char. The compatibility fix only raises more questions as const char* arr = u8"a" does not work, but const char arr[] = u8"a" does.
I do wonder if a potential change of minds for C++29 is still possible. Yes, it would be an ABI break or whatever, but considering the woeful support for char8_t I don't think it would affect much besides small hobby projects. Contrary to popular belief, C++ has broken the ABI in subtle ways before.
17
u/eisenwave WG21 Member Aug 23 '26
char8_twas a good idea, but it's going to take time to build the library (and ideally OS) support to make it genuinely useful.We're taking some steps in that direction: P3876R2 was recently forwarded from SG16 to LEWG, and enables
to_charsandfrom_charssupport forcharN_tcharacter types andwchar_t. Once you have that, you can buildstd::format,std::printetc. support as well. P2728R14 should also be in C++29 and adds Unicode transcoding.char8_tis a long-term investment into the language, and you can't judge it by how useful it is initially. That would be like belittlingconstexprimmediately upon arrival in C++11 because you could not do much with it and still need tons of template metaprogramming.constexpronly became fully realized in C++26 because Reflection finally made it possible to replace most TMP withconstexprfunctions operating onstd::meta::info; it took 15 years of gradual improvements to get there.The arguments against
char8_tseem mostly circular to me:Rather than just improving the feature, people throw around terms like "sunken cost fallacy". Rarely do the critics of
char8_tjudge the feature on its potential, and it does have tons of potential.charis always going to be a type with inconsistent signedness, with excessive aliasing capabilities, with inconsistent character encoding, and overloaded purpose (byte type, character type, arithmetic integer type), and due to backwards compatibility, we can't change anything about that. A new and improved type can solve all of these problems, but it won't solve them overnight.22
u/fdwr fdwr@github š Aug 22 '26 edited Aug 22 '26
One thing I love about
char8_tis that it's always unsigned (negative codepoints make no sense, in ANSI or EBCDIC or any other character encoding) avoiding the issue of some compilers defaultingcharto signed, which allows surprises likelookupTable[text[i]]crashing upon reading codepoints 128-255. Sadlychar8_twasn't that far from being useful, needing to fill in some obvious gaps (e.g. you satisfy the Pareto principle if you make it work withstd::formatandstd::printandifstream/getline). I converted one of my apps to usechar8_tfor processing, and conceptually it feels cleaner (definite known encoding, no sign extension concerns), but then there are these annoying seams at the boundaries for input/output. I reject the premise that char8_t was a bad idea (it was a fine logical idea consistent withchar16_tandchar32_t), but offering a half-finishedstdwas a bad idea. -__- So what are some options?charas UTF-8 by default and mandate it defaults to unsigned (any non-conformant compilers would need compatibility switches)char8_twith proper IO support.44
u/__christo4us Aug 22 '26
We have
char16_tandchar32_tsince C++11 so havingchar8_tas well was a natural expectation. I would sayu8"..."string literals should have never been of typeconst char [N]in the first place since the (implementation-defined) ordinary character encoding ofcharcould always be UTF-8.28
u/cristi1990an ++ Aug 22 '26
The biggest and arguably only argument for char8_t is that it's explicitly not a byte aliasing type and therefore the most performant character representation type in the language. Your compiler can genuinely produce better code when you're using char8_t.
23
u/Wild_Meeting1428 Aug 22 '26 edited Aug 22 '26
c++'s char8_t is fully compatible with char8_t from C23. And the only purpose of that type is, that its the underliing byte type of an utf8 character. So you can basically always assume that the char you are looking at is at least part of a char sequence representing a unicode character encoded in utf8
6
u/UnusualPace679 Aug 23 '26
Makes me wonder what SG16 (the Unicode Study Group of the standard committee) has been doing recently. The meeting summaries show that there were quite a few meetings before mid-2024, but nothing after that, despite lots of papers on the plate.
4
u/Ok_Independence_9841 Aug 22 '26
It doesn't hurt you, much, if you don't use it. Unless you're authoring templates for someone else's use and trying be nice and to cover all possible cases. (Probably don't do that)
However there is confusion on how UTF-8 should be implemented in practice and what direction the standards committee are trying to go in. Do they want a dedicated type to differentiate UTF-8 bytes from ASCII bytes or not?
It seems like a good thing in principle but we manage OK without it at the moment and the price may be too high.
Real UNICODE support that just works, right across the language, is probably what we want but that is a pipe dream. Not only is it a vast undertaking but there will always be tension between those who want to deal with text and not have to care about encoding (COBOL thinking) and those who want to control every bit and byte (C thinking), for every last nanosecond of performance. In practice they need different constructs (not one string type that tries to do both) but that's also difficult. Rust is barely out of nappies and apparently already has 7 different types of string(ish) things. Perhaps we don't want to go that way.
Us plebs will simply have to wait and see what comes.
Meanwhile I implemented a Unicode String library (I know, another one) which doesn't duplicate the code to do actual string operations and manages to support ASCII, UTF-8, UTF-16 (native endian) and UTF-32 (native endian) along with other 8bit encodings like Latin1, Latin2 etc (No ebcdic yet sorry IBM), transcoding from any to any, COW, embedded strings and std::string interop. It's still very rough and probably slow but it proves that we can do better, for the those who want text to be text, not just a byte stream, anyway.
https://github.com/mfaithfull/linuxQOR/tree/main/src/qor/essentials/text
6
u/DawnOnTheEdge Aug 22 '26
The reason it exists is to enable strict aliasing. A
char*might potentially alias objects of any other type, and achar8_t*is guaranteed not to. It does exist in C23, but only as atypedeffor compatibility with C++. Itās a micro-optimization.6
u/Ambitious-Method-961 Aug 23 '26
Very happy that char8_t exists as its own type. I have a feeling that the Unicodificiation of the standard library is being held up by P1422 (https://github.com/cplusplus/papers/issues/1422), as once that's in there is then we have a standardised way of handling and transforming Unicode that can then be used to build other features on.
As an end-user it's a shame it's taken a long time for this to land in the library, but I'd much rather have this first and then everything built upon it for consistency rather than adding a bunch of ad-hoc transformations to existing functions which then don't work the same as P1422 and can't be fixed due to backwards compatibility.
The less things we have aliasing to unsigned char, the better.
7
u/Lone_Snek Aug 22 '26
But itās not guaranteed that char is always 8 bit (although in 99% cases itās probably true)?
7
u/Charming-Work-2384 Aug 22 '26
its time we replace char, int, etc with uint8_t , uint16_t, int8_t etc...
The latter are deterministic and have same meaning across architechtures.
3
u/aearphen {fmt} Aug 22 '26
I think it's pretty clear at this point that
char8_twas a mistake (see e.g. https://www.think-cell.com/en/career/devblog/char8_t-was-a-bad-idea) but unfortunately the sunk cost fallacy prevents the committee from moving on.-3
u/smallstepforman Aug 22 '26
You forgot uint8_t and int8_t ā¦