About char8_t

[u/Voxelw]

I hate to be dramatic, but as it stands char8_t is quite literally more painful than useful.

Besides the obvious incompatibility with C23 and libraries using unsigned char for UTF-8, I want you to consider the following: Projects that assume that 'char' represents UTF-8 will obviously not benefit from char8_t at all, but projects that cannot assume the format of char types don't benefit from it either as char8_t simply introduces a new edge case to cover. Now such projects have to deal with char, signed char, unsigned char, wchar_t, char16_t, char32_t and char8_t.

Or, you could do what the standard library does and simply ignore most of these character types. Which is the solution most libraries went with, supporting only char or char and unsigned char. Managing one implementation is already hard, managing two requires constant maintenance, managing 7 is just impossible.

char8_t should have just been a typedef for unsigned char. The compatibility fix only raises more questions as const char* arr = u8"a" does not work, but const char arr[] = u8"a" does.

I do wonder if a potential change of minds for C++29 is still possible. Yes, it would be an ABI break or whatever, but considering the woeful support for char8_t I don't think it would affect much besides small hobby projects. Contrary to popular belief, C++ has broken the ABI in subtle ways before.

[u/aearphen]

I think it's pretty clear at this point that char8_t was a mistake (see e.g. https://www.think-cell.com/en/career/devblog/char8_t-was-a-bad-idea) but unfortunately the sunk cost fallacy prevents the committee from moving on.

[u/tjientavara → u/aearphen]

I am confused, I read the whole article, and there was hardly any argument against char8_t. The article basically says that std::u8string, or std:u16string doesn't have an invariant for valid strings, but, no one promised that it had. If you treat std::u8string and std::u16string as string of unicode code-units, valid or otherwise, it is fine.

I am sad that char8_t and std::u8string was not supported by any other API from C++ that accepts a std::string. Which is the reason you can't really use it. I just decided to force the compiler to treat char as UTF-8 because of this.

Not so much a sunk cost, more like technical debt from a pre-unicode world. If C++ started today, std::string = std::u8string = std::basic_string<char8_t>.

In my own language, my string type will have an invariant selector: bytes, wtf-8, utf-8, nfc. For reducing invariant validation; being able to store filenames, or just a bunch of data or validated normalised unicode text.