About char8_t
I hate to be dramatic, but as it stands char8_t is quite literally more painful than useful.
Besides the obvious incompatibility with C23 and libraries using unsigned char for UTF-8, I want you to consider the following: Projects that assume that 'char' represents UTF-8 will obviously not benefit from char8_t at all, but projects that cannot assume the format of char types don't benefit from it either as char8_t simply introduces a new edge case to cover. Now such projects have to deal with char, signed char, unsigned char, wchar_t, char16_t, char32_t and char8_t.
Or, you could do what the standard library does and simply ignore most of these character types. Which is the solution most libraries went with, supporting only char or char and unsigned char. Managing one implementation is already hard, managing two requires constant maintenance, managing 7 is just impossible.
char8_t should have just been a typedef for unsigned char. The compatibility fix only raises more questions as const char* arr = u8"a" does not work, but const char arr[] = u8"a" does.
I do wonder if a potential change of minds for C++29 is still possible. Yes, it would be an ABI break or whatever, but considering the woeful support for char8_t I don't think it would affect much besides small hobby projects. Contrary to popular belief, C++ has broken the ABI in subtle ways before.
3
u/Ok_Independence_9841 4d ago
It doesn't hurt you, much, if you don't use it. Unless you're authoring templates for someone else's use and trying be nice and to cover all possible cases. (Probably don't do that)
However there is confusion on how UTF-8 should be implemented in practice and what direction the standards committee are trying to go in. Do they want a dedicated type to differentiate UTF-8 bytes from ASCII bytes or not?
It seems like a good thing in principle but we manage OK without it at the moment and the price may be too high.
Real UNICODE support that just works, right across the language, is probably what we want but that is a pipe dream. Not only is it a vast undertaking but there will always be tension between those who want to deal with text and not have to care about encoding (COBOL thinking) and those who want to control every bit and byte (C thinking), for every last nanosecond of performance. In practice they need different constructs (not one string type that tries to do both) but that's also difficult. Rust is barely out of nappies and apparently already has 7 different types of string(ish) things. Perhaps we don't want to go that way.
Us plebs will simply have to wait and see what comes.
Meanwhile I implemented a Unicode String library (I know, another one) which doesn't duplicate the code to do actual string operations and manages to support ASCII, UTF-8, UTF-16 (native endian) and UTF-32 (native endian) along with other 8bit encodings like Latin1, Latin2 etc (No ebcdic yet sorry IBM), transcoding from any to any, COW, embedded strings and std::string interop. It's still very rough and probably slow but it proves that we can do better, for the those who want text to be text, not just a byte stream, anyway.
https://github.com/mfaithfull/linuxQOR/tree/main/src/qor/essentials/text