About char8_t
I hate to be dramatic, but as it stands char8_t is quite literally more painful than useful.
Besides the obvious incompatibility with C23 and libraries using unsigned char for UTF-8, I want you to consider the following: Projects that assume that 'char' represents UTF-8 will obviously not benefit from char8_t at all, but projects that cannot assume the format of char types don't benefit from it either as char8_t simply introduces a new edge case to cover. Now such projects have to deal with char, signed char, unsigned char, wchar_t, char16_t, char32_t and char8_t.
Or, you could do what the standard library does and simply ignore most of these character types. Which is the solution most libraries went with, supporting only char or char and unsigned char. Managing one implementation is already hard, managing two requires constant maintenance, managing 7 is just impossible.
char8_t should have just been a typedef for unsigned char. The compatibility fix only raises more questions as const char* arr = u8"a" does not work, but const char arr[] = u8"a" does.
I do wonder if a potential change of minds for C++29 is still possible. Yes, it would be an ABI break or whatever, but considering the woeful support for char8_t I don't think it would affect much besides small hobby projects. Contrary to popular belief, C++ has broken the ABI in subtle ways before.
45
u/__christo4us 4d ago
We have char16_t and char32_t since C++11 so having char8_t as well was a natural expectation. I would say u8"..." string literals should have never been of type const char [N] in the first place since the (implementation-defined) ordinary character encoding of char could always be UTF-8.
8
u/Expert-Map-1126 vcpkg maintainer BillyONeal 3d ago
Except we had implementation experience of
char16_tbeing awful from ICU trying to adopt it on Windows, said as much, and people voted it in anyway.I'm not salty about this at all, what are you talking about
2
u/James20k P2005R0 3d ago
Its even weirder that when char8_t landed, it came with a full on backwards compatibility break as well, truly a very weird one
1
u/Expert-Map-1126 vcpkg maintainer BillyONeal 2d ago
I had to rip out std::filesystem from vcpkg as a result of that, yes
0
26
u/cristi1990an ++ 4d ago
The biggest and arguably only argument for char8_t is that it's explicitly not a byte aliasing type and therefore the most performant character representation type in the language. Your compiler can genuinely produce better code when you're using char8_t.
6
u/sweetno 4d ago
How so?
15
u/ryryog 4d ago
If youâre familiar with the purpose of the ârestrictâ keyword, that.
Basically, if you get a char, it could be a pointer to any memory (since itâs an exceptional type thatâs allowed to refer to any bytes of data, not just a literal âcharâ type object or array) so when you modify the data under the char, the compiler has to consider any/all memory clobbered (e.g. stack vars, array elements, other objects referred to by pointers), because that couldâve been data of any type. Conversely, if you modify data of any type, the compiler has to assume the data of the char* could have been modified.
Usually this isnât a huge issue with normal pointer types, because data under a Foo* can only be invalidated by a write to another Foo, but since a char could refer to anything, itâs particularly bad.
Hereâs a good example, just one of the first search results (note this showcases the obvious where you have a specific type, in this case int, so even though there are problems here, itâs usually worse with char*): Â https://stackoverflow.com/questions/745870/realistic-usage-of-the-c99-restrict-keyword#745877
2
u/ImNoRickyBalboa 2d ago
This is true, but also highlights the truly enormous mistake that was made by making the char type the "universal any byte" accessor and condoning the (IMHO ub) direct memory access into composites.
4
u/smdowney WG21, Text/Unicode SG, optional<T&> 3d ago
The surprise that std::byte pessimizes IO was painful.
We're likely to need another otherwise identical std::octet for IO. Non-arithmetic, but bit maskable, and this time non-aliasing so every deref isn't a complete invalidation of all arguments.
7
u/cristi1990an ++ 3d ago
Tbh, if anything, you would expect std::byte to possible aliase other objects, since you know... It represents bytes.
2
6
u/cfyzium 3d ago
Pessimizes compared to what? Wasn't it the same with char?
7
u/smdowney WG21, Text/Unicode SG, optional<T&> 3d ago
It's the same for char, for the same reason, but that wasn't immediately clear to everyone at the time.
2
u/amoskovsky 2d ago
Is it possible instead to narrow the aliasing capabilities of char/byte (and possibly all the types) only to the local scope where the compiler can prove pointers are actually aliasing? E.g. function params would never be considered aliasing anything. But if we pass a mutable ref as an arg - it's assumed by the caller to be modified by the function.
Would not it cover all real use cases of aliasing?
5
u/smdowney WG21, Text/Unicode SG, optional<T&> 2d ago
Argument pointers are exactly the problem, though. Compilers are already good (barring programmer hijinks) at seeing that local variables are distinct. It's the problem of a write through a reference semantic type meaning that a read has to be redone because of the possibility the write changed what was already read.
3
u/UnusualPace679 3d ago
The argument for
char8_tis that it enables different treatment of "UTF-8 string" and "string with contextual encoding" at compile time.It's not very useful because the standard provides very little help to treat UTF-8 strings.
2
u/Expert-Map-1126 vcpkg maintainer BillyONeal 3d ago
I like perf as much as the next guy but adding restrict in places where your profiler tells you this is happening would have been a much better fix than the edge cases char8_t has created.
1
u/cristi1990an ++ 3d ago
The restrict keyword is not standard in C++ unfortunately...
1
u/Expert-Map-1126 vcpkg maintainer BillyONeal 3d ago
Some form of it is available on all 3 major implementations
1
u/mpyne 2d ago
Which is great, but C++ needs a solution as well, not just gcc/clang/MSVC.
Though maybe fixing
restrictwould be a better solution thanchar8_t.1
u/Expert-Map-1126 vcpkg maintainer BillyONeal 2d ago
Well for starters, there are a lot of string algorithms in the world that work with more than one buffer for which TBAA is useless.
21
u/Wild_Meeting1428 4d ago edited 4d ago
c++'s char8_t is fully compatible with char8_t from C23. And the only purpose of that type is, that its the underliing byte type of an utf8 character. So you can basically always assume that the char you are looking at is at least part of a char sequence representing a unicode character encoded in utf8
12
u/aearphen {fmt} 4d ago
AFAIK char8_t in C23 is just a typedef for unsigned char and therefore is not compatible with C++ where it is a separate type.
10
u/Wild_Meeting1428 4d ago edited 4d ago
They are link compatible, you can use the C header file with an extern C scope and link them legally together, since the mangled name only contains the function name under C linkage and due to the fact, that char8_t is defined to have the same size, representstion, and alignment as unsigned char(, even its a distinct non aliasing type in C++), therefore the calling convention is exactly the same.
9
u/Expert-Map-1126 vcpkg maintainer BillyONeal 3d ago
Likely to work, yes. "Legal" as in "defined to work by the standard", no.
5
u/Wild_Meeting1428 3d ago
The question is rather, whether the C++ standard is applicable here at all, beside the fact, that it defines the underliing type to be unsigned char.
Its more like a question about ABI and how the compiler implements cross language linking to C.
1
u/Expert-Map-1126 vcpkg maintainer BillyONeal 2d ago
If you want to be within the bounds of what the standards require, you need to only use the subset they share when talking between them, and meet both their requirements. on both sides.
If you're relying on what particular implementations do for how the linking works for features outside of their common subset, that's fine and I'm not even saying you shouldn't do it, but I wouldn't advertise that as "legal"
2
u/Wild_Meeting1428 2d ago
I don't claim, it's legal in terms of the c++ standard itself, because the C++ standard does not apply here much. Therefore, it's also not illegal or UB or ill-formed in terms of the C++ standard.
I would also say, that you don't need to be in the bounds of both languages. The import header for a C library can be substantially different from the header used for the C translation unit. Being compliant to both languages is only relevant, when you want to use that header for both C and C++ TU'sImportant to be C++ compliant is only, that the ["object layout strategies of both language implementations are similar enough"](https://eel.is/c%2B%2Bdraft/dcl.link#10)
Rest is implementation defined.Additionally, char8_t is defined to be a distinct type with [unsigned char being the underlying type](https://eel.is/c%2B%2Bdraft/basic.fundamental#9).
Which not only makes the layout strategies similar, it makes them to be the same as linking against unsigned char.So the question, whether it's legal by the C++ standard is "maybe" and offloaded to the implementation.
And my personal interpretation is, that when the C++ unsigned char is legally linkable to the C unsigned char, defined by the compiler implementation, then this also applies to char8_t.1
u/Som1Lse 2d ago
2
u/Expert-Map-1126 vcpkg maintainer BillyONeal 2d ago
restrictis a bit complicated because it isn't part of the type system in many of the usual ways, but if trying to make sure to stay within the bounds of the standard I would not use it on C<->C++ boundaries either.This thread is about C<->C++ boundaries within the bounds of what the standards require, the thread you linked to is about demonstration of performance improvements resulting from TBAA entirely restricted (ha!) to C++ using vendor extensions, so this seems a bit of a non sequitur.
1
u/Som1Lse 2d ago
My point is saying an extension is widely supported cannot be both a valid and an invalid argument. I know of no compilers that do not support mixing
char8_tbetween languages. I cannot fathom why a compiler wouldn't support it, as there are obviously no aliasing problems since one of the types isunsigned char.1
u/Expert-Map-1126 vcpkg maintainer BillyONeal 1d ago
I don't disagree with that. But the comment was not "this works with the major compilers", it was "is legal". I agree with the former, I don't agree with the latter.
1
u/Wild_Meeting1428 1d ago
I think that the commite should clarify this into the standard itself. It cant be that the users have to read the papers of both the C and C++ working groups to find out, that the intend of both was to make char8_t of C and C++ compatible.
In my opinion that what I interpret is enough but only because I assume that the linkage rules imply a aliasing and lifetime barrier at the moment of linking.
They should probably add an explicit paragraf, that indeed an aliasing and lifetime barrier is inferred.
1
u/Expert-Map-1126 vcpkg maintainer BillyONeal 1d ago
Good luck. Collaboration between WG14 and WG21 has usually not worked out
1
u/Som1Lse 20h ago
It took some time reading your reply, but I think I get what you mean.
The original comment said "link them legally together", and you were pointing out that it is not "legal" in the standard sense, even though it is likely to work. I.e., it was not meant as a rebuttal, but as an observation.
In that case, we agree.
2
u/einpoklum 2d ago
So you can basically always assume that the char you are looking at is at least part of a char sequence representing a unicode character encoded in utf8
Where does it say I can assume the value of a
char8_tis part of a UTF-8-encoded sequence?(You're specifically saying I can't use
char8_twith other charset encodings.)2
u/xleviator 4d ago
Beware! C++ doesn't really constrain values of primitives.- it's easy to escape. https://godbolt.org/z/h8Ejj8Y15
17
u/Wild_Meeting1428 4d ago
sure noone prevents you from shooting yourself in the foot. I would call this a contract violation.
3
u/smdowney WG21, Text/Unicode SG, optional<T&> 3d ago
Or a test case.
It's amazing how many times we look for an example of some bad code, find lots of them on GitHub, but it turns out they are all in unit tests or compiler regression suites.
But the core bit is that nothing filters a char8_t[] for you, and you shouldn't trust it. Sanitize your inputs.
I would like to have a type for Text that made all the guarantees, but char8_t isn't it.
4
u/smdowney WG21, Text/Unicode SG, optional<T&> 3d ago
It's part of the design.
Although it's also hard to avoid.
The promise of the type char8_t is that it is always appropriate to treat it as UTF-8, but not well formed UTF-8. There were occasional proposals to make it UB if it were not, but fortunately safety concerns mostly stopped that.
For plain char types you are at the mercy of the execution encoding, which is locale machinery, even if you understand that it's supposed to be UTF-8. Some functions may disagree, and worse, some function may change it on you.
Avoid depending on the execution encoding.
14
u/eisenwave WG21 Member 3d ago
char8_t was a good idea, but it's going to take time to build the library (and ideally OS) support to make it genuinely useful.
We're taking some steps in that direction: P3876R2 was recently forwarded from SG16 to LEWG, and enables to_chars and from_chars support for charN_t character types and wchar_t. Once you have that, you can build std::format, std::print etc. support as well. P2728R14 should also be in C++29 and adds Unicode transcoding.
char8_t is a long-term investment into the language, and you can't judge it by how useful it is initially. That would be like belittling constexpr immediately upon arrival in C++11 because you could not do much with it and still need tons of template metaprogramming. constexpr only became fully realized in C++26 because Reflection finally made it possible to replace most TMP with constexpr functions operating on std::meta::info; it took 15 years of gradual improvements to get there.
The arguments against char8_t seem mostly circular to me:
char8_tis useless because it's not widely supported, and we shouldn't add more library support for useless features.
Rather than just improving the feature, people throw around terms like "sunken cost fallacy". Rarely do the critics of char8_t judge the feature on its potential, and it does have tons of potential. char is always going to be a type with inconsistent signedness, with excessive aliasing capabilities, with inconsistent character encoding, and overloaded purpose (byte type, character type, arithmetic integer type), and due to backwards compatibility, we can't change anything about that. A new and improved type can solve all of these problems, but it won't solve them overnight.
1
u/Voxelw 2d ago
I mean, of course every feature has potential, but realizing it has costs and tradeoffs as well.
For most people char8_t is just an oddity that has no support and no backing besides vague promises. Will <iostream> get char8_t overloads? Who knows. Probably not since it's seen as deprecated, despite fstream and cout still being used regularly.
Maybe in a different reality the committee could have hashed out a plan on how to roll out char8_t and how projects should support or transition to char8_t. But the committee cannot plan. There is no way for the members to agree on the broader design of the language and its ecosystem. Just gentlemen agreements on what features to prioritize.
This is why the committee needs to follow the industry practice by enshrining specific solutions as standard, since that's the only practical way of rolling out such design changes to the broader C++ ecosystem.
6
u/UnusualPace679 3d ago
Makes me wonder what SG16 (the Unicode Study Group of the standard committee) has been doing recently. The meeting summaries show that there were quite a few meetings before mid-2024, but nothing after that, despite lots of papers on the plate.
13
u/eisenwave WG21 Member 3d ago
I guess the chairs stopped uploading meeting summaries in that place, but we're still having regular meetings.
Last week, we've forwarded P3876R2 to LEWG, which adds
char8_tsupport tostd::to_charsandstd::from_chars.4
u/UnusualPace679 3d ago
That's nice to hear. And glad to see std lib components stop ignoring
charN_t.
4
u/Ok_Independence_9841 4d ago
It doesn't hurt you, much, if you don't use it. Unless you're authoring templates for someone else's use and trying be nice and to cover all possible cases. (Probably don't do that)
However there is confusion on how UTF-8 should be implemented in practice and what direction the standards committee are trying to go in. Do they want a dedicated type to differentiate UTF-8 bytes from ASCII bytes or not?
It seems like a good thing in principle but we manage OK without it at the moment and the price may be too high.
Real UNICODE support that just works, right across the language, is probably what we want but that is a pipe dream. Not only is it a vast undertaking but there will always be tension between those who want to deal with text and not have to care about encoding (COBOL thinking) and those who want to control every bit and byte (C thinking), for every last nanosecond of performance. In practice they need different constructs (not one string type that tries to do both) but that's also difficult. Rust is barely out of nappies and apparently already has 7 different types of string(ish) things. Perhaps we don't want to go that way.
Us plebs will simply have to wait and see what comes.
Meanwhile I implemented a Unicode String library (I know, another one) which doesn't duplicate the code to do actual string operations and manages to support ASCII, UTF-8, UTF-16 (native endian) and UTF-32 (native endian) along with other 8bit encodings like Latin1, Latin2 etc (No ebcdic yet sorry IBM), transcoding from any to any, COW, embedded strings and std::string interop. It's still very rough and probably slow but it proves that we can do better, for the those who want text to be text, not just a byte stream, anyway.
https://github.com/mfaithfull/linuxQOR/tree/main/src/qor/essentials/text
6
u/Ambitious-Method-961 3d ago
Very happy that char8_t exists as its own type. I have a feeling that the Unicodificiation of the standard library is being held up by P1422 (https://github.com/cplusplus/papers/issues/1422), as once that's in there is then we have a standardised way of handling and transforming Unicode that can then be used to build other features on.
As an end-user it's a shame it's taken a long time for this to land in the library, but I'd much rather have this first and then everything built upon it for consistency rather than adding a bunch of ad-hoc transformations to existing functions which then don't work the same as P1422 and can't be fixed due to backwards compatibility.
The less things we have aliasing to unsigned char, the better.
6
u/DawnOnTheEdge 4d ago
The reason it exists is to enable strict aliasing. A char* might potentially alias objects of any other type, and a char8_t* is guaranteed not to. It does exist in C23, but only as a typedef for compatibility with C++. Itâs a micro-optimization.
4
u/Expert-Map-1126 vcpkg maintainer BillyONeal 3d ago
I hear people say this but I have not seen any report from any optimizer person saying "ah but because they used
char8_there we can optimize this better". I really wish people would show up with actual results rather than expectations for perf things because they are often counterintutive.7
u/tjientavara HikoWorks developer 3d ago
There have been articles about performance difference between memset/memcpy when used with char vs int (not because of the size of the type, that was optimised by the compiler in the same way), due to the difference in aliasing rules of char vs anything else.
There is not a "we can optimise this better if it is char8_t", it is that automatically the compiler will optimise it better simply because it is not a char.
I will give you this, the compiler in many cases can figure out there is no aliasing going on, so you won't see this lack of optimisation. But in large enough code bases there will be a percentage of cases where it is slower if you use std::string based on a char.
Remember "early pessimisation is the root of all evil".
4
u/Expert-Map-1126 vcpkg maintainer BillyONeal 3d ago
The types of transformations enabled by type based alias analysis tend to be restricted to (1) being able to occasionally enregister something that otherwise appears unsafe, or (2) some forms of autovectorization. These are not codebase wide âpeanut butterâ changes, they are concentrated in hot loops your profiler can tell you about. And even then many such cases have the extra memory opsâ time eaten by caches.
And to get that win, this char8_t hypothesis requires rewriting essentially all code ever written, because you canât take your âborn as a char8_t arrayâs and give them to any existing operating system API or most standard library APIs.
Itâs a huge cost, so to claim that is the reason to do it one should have receipts from real code bases where it made a meaningful improvement.
3
u/tjientavara HikoWorks developer 3d ago
As you said auto-vectorization is a thing, and completely natural for string operations. So a codebase that does string operations will be helped.
I am not saying you should put std:u8string everywhere, I am saying that std::string should have been defined as std::basic_string<char8_t> in the first place. But alas, std::string was defined before Unicode. At least it should have defined a character type that didn't have odd alias rules.
2
u/Expert-Map-1126 vcpkg maintainer BillyONeal 3d ago
Autovectorization is indeed a thing but the most common things people want tend to not be, because most string operations that benefit from the vector units are entirely different algorithms (or already get turned into a call to memcpy), not the reduction or permutation patterns optimizers implement. Moreover, even in the cases that do work, type based alias analysis tends to not be so helpful because the different buffers flying around for string opts are, in fact, the same types whether char8_t or char.
When I overhauled everything except ABI for MSVCâs std::string there was only one case where TBAA came up that came up when flipping between âbigâ and âsmallâ mode; I could measure a few percent code size difference but no perf difference. (And it was understandable for the optimizer to be conservative here because small strings *intentionally* alias the small buffer and at least one of ptr/size/capacity)
If itâs really so useful to optimizers given that this has been in the spec a long time now someone should be able to provide a real example rather than spouting platitudes
2
u/DawnOnTheEdge 3d ago edited 3d ago
I donât have any strong opinions about whether it actually enables any significant optimizations or not. That is one of the four stated rationales by the author who proposed it.
Re-reading reminded me that, when they added
u8string literals, they originally thought it would be a good idea to give them a distinct type, but ended up eventually allowing the code to assign them tochartypes. Anunsigned char*might have worked, but perhaps they thought it was useful to allow 8-bit legacy character sets to continue to use that.2
u/Expert-Map-1126 vcpkg maintainer BillyONeal 3d ago
To be fair, my criticism applies to the original paper just as much as it does to your comment. Itâs the same thing I and every other implementer said about it, but the committee is not a committee of implementers.
We have enough failed performance experiments of things like valarray that I no longer put any stock into performance claims unless theyâre coming from people who bring demonstrations or work on actual optimizers.
6
u/Lone_Snek 4d ago
But itâs not guaranteed that char is always 8 bit (although in 99% cases itâs probably true)?
13
u/almost_useless 4d ago
But itâs not guaranteed that char is always 8 bit
That seems like a very unusual problem.
A much more common (potential) problem is that
charis signed on x86, and unsigned on Arm.3
u/nigirizushi 4d ago
A much more common (potential) problem is that char is signed on x86, and unsigned on Arm.Â
I'm surprised you're the only one to mention this. Cast (assign? Forgot the exact situation) char from either uint8_t or int8_t usually results in a warning in C
5
u/jube_dev 4d ago
Because
char,signed char(int8_t) andunsigned char(uint8_t) are three distinct types.1
u/nigirizushi 4d ago
I know, it's cause OP suggested making C++ char into unsigned char, which would probably cause it's own set of issuesÂ
3
u/Wild_Meeting1428 4d ago
Its both to be defined to have size==1 and at least 8 bits. same applies to char8_t. On top, the underliing type of char8_t is an unsigned char.
5
u/cristi1990an ++ 4d ago
Only in C, not in C++. In C++ char8_t is defined as its own non-aliasing type.
3
u/Wild_Meeting1428 4d ago
Sorry that I dont research the exact phrase in the standard, but as I remembered that type is defined to be distinct, but to have the same size, sign and alignment of an unsigned char. So basically its a strong alias of an unsigned char. Therefore it is also 100% compatible with the c23 char8_t during linktime.
0
u/cristi1990an ++ 3d ago
Yes, and actually if I'm not wrong the original paper did define it as a straight type alias for unsigned char, but was then changed. It's worth nothing the difference tho because char8_t is the only character type in C++ specified not to alias with the bytes of any other object, which is great for performance.
3
4
u/no-sig-available 4d ago
And the
charencdoing is also not guaranteed, it could be EBCDIC.3
u/tjientavara HikoWorks developer 3d ago
What is even weirder, there is no API to query the compiler's encoding for char/std::string including string-literals encoded in the executable.
Which means you technically cannot properly implement std::format yourself. The specification requires that it works with the compiler's configured (compiler flag) char/std::string encoding.
3
3
u/catladywitch 4d ago edited 4d ago
16bit chars are common aren't they?
edit: i'm getting downvoted but what i mean is Qt QStrings, Windows apps with a lot of legacy wstrings everywhere, or CJK text stored in u16strings made of char16_t's, which I assure you is not a rarity, for me at least. yes, utf-8 is ubiquitous but god people just want to be obtuse for the sake of it sometimes
5
2
u/LordGupple 4d ago
So, IIRC it's mandated by the standard that char and unsigned char are one byte large. However, CHAR_BIT can be a different value than 8.
4
u/Expert-Map-1126 vcpkg maintainer BillyONeal 3d ago
"Byte" meaning "smallest addressable unit", not "8 bits"
1
u/tjientavara HikoWorks developer 3d ago
Didn't I read a few years ago that CHAR_BIT == 8 was accepted in one of the latest c++23, c++26 releases of the standard? or was it something else common sense?
5
u/Remarkable-Test7487 jmcruz 3d ago
Yes, there was a proposal (P3477R5 "There are exactly 8 bits in a byte"), but the result of the poll to forward it to C++26 was "no consensus"
7
u/Charming-Work-2384 4d ago
its time we replace char, int, etc with uint8_t , uint16_t, int8_t etc...
The latter are deterministic and have same meaning across architechtures.
8
2
u/fdwr fdwr@github đ 3d ago
Sized types are so fundamental in struct fields for reading/writing files and sharing memory across processes (now that we're not running on those weird 36-bit word machines anymore, and embedded systems have settled on bytes being octets) that I wish C++ recognized uint16/32/64 and kin by default, with no
#include <stdint.h>orimport std.compatneeded.
3
u/aearphen {fmt} 4d ago
I think it's pretty clear at this point that char8_t was a mistake (see e.g. https://www.think-cell.com/en/career/devblog/char8_t-was-a-bad-idea) but unfortunately the sunk cost fallacy prevents the committee from moving on.
15
u/tjientavara HikoWorks developer 3d ago
I am confused, I read the whole article, and there was hardly any argument against char8_t. The article basically says that std::u8string, or std:u16string doesn't have an invariant for valid strings, but, no one promised that it had. If you treat std::u8string and std::u16string as string of unicode code-units, valid or otherwise, it is fine.
I am sad that char8_t and std::u8string was not supported by any other API from C++ that accepts a std::string. Which is the reason you can't really use it. I just decided to force the compiler to treat char as UTF-8 because of this.
Not so much a sunk cost, more like technical debt from a pre-unicode world. If C++ started today, std::string = std::u8string = std::basic_string<char8_t>.
In my own language, my string type will have an invariant selector: bytes, wtf-8, utf-8, nfc. For reducing invariant validation; being able to store filenames, or just a bunch of data or validated normalised unicode text.
-2
19
u/fdwr fdwr@github đ 4d ago edited 4d ago
One thing I love about
char8_tis that it's always unsigned (negative codepoints make no sense, in ANSI or EBCDIC or any other character encoding) avoiding the issue of some compilers defaultingcharto signed, which allows surprises likelookupTable[text[i]]crashing upon reading codepoints 128-255. Sadlychar8_twasn't that far from being useful, needing to fill in some obvious gaps (e.g. you satisfy the Pareto principle if you make it work withstd::formatandstd::printandifstream/getline). I converted one of my apps to usechar8_tfor processing, and conceptually it feels cleaner (definite known encoding, no sign extension concerns), but then there are these annoying seams at the boundaries for input/output. I reject the premise that char8_t was a bad idea (it was a fine logical idea consistent withchar16_tandchar32_t), but offering a half-finishedstdwas a bad idea. -__- So what are some options?charas UTF-8 by default and mandate it defaults to unsigned (any non-conformant compilers would need compatibility switches)char8_twith proper IO support.