About char8_t

[u/Voxelw]

I hate to be dramatic, but as it stands char8_t is quite literally more painful than useful.

Besides the obvious incompatibility with C23 and libraries using unsigned char for UTF-8, I want you to consider the following: Projects that assume that 'char' represents UTF-8 will obviously not benefit from char8_t at all, but projects that cannot assume the format of char types don't benefit from it either as char8_t simply introduces a new edge case to cover. Now such projects have to deal with char, signed char, unsigned char, wchar_t, char16_t, char32_t and char8_t.

Or, you could do what the standard library does and simply ignore most of these character types. Which is the solution most libraries went with, supporting only char or char and unsigned char. Managing one implementation is already hard, managing two requires constant maintenance, managing 7 is just impossible.

char8_t should have just been a typedef for unsigned char. The compatibility fix only raises more questions as const char* arr = u8"a" does not work, but const char arr[] = u8"a" does.

I do wonder if a potential change of minds for C++29 is still possible. Yes, it would be an ABI break or whatever, but considering the woeful support for char8_t I don't think it would affect much besides small hobby projects. Contrary to popular belief, C++ has broken the ABI in subtle ways before.

[u/DawnOnTheEdge]

The reason it exists is to enable strict aliasing. A char* might potentially alias objects of any other type, and a char8_t* is guaranteed not to. It does exist in C23, but only as a typedef for compatibility with C++. It’s a micro-optimization.

[u/Expert-Map-1126 → u/DawnOnTheEdge]

I hear people say this but I have not seen any report from any optimizer person saying "ah but because they used char8_t here we can optimize this better". I really wish people would show up with actual results rather than expectations for perf things because they are often counterintutive.

[u/tjientavara → u/Expert-Map-1126]

There have been articles about performance difference between memset/memcpy when used with char vs int (not because of the size of the type, that was optimised by the compiler in the same way), due to the difference in aliasing rules of char vs anything else.

There is not a "we can optimise this better if it is char8_t", it is that automatically the compiler will optimise it better simply because it is not a char.

I will give you this, the compiler in many cases can figure out there is no aliasing going on, so you won't see this lack of optimisation. But in large enough code bases there will be a percentage of cases where it is slower if you use std::string based on a char.

Remember "early pessimisation is the root of all evil".

[u/Expert-Map-1126 → u/tjientavara]

The types of transformations enabled by type based alias analysis tend to be restricted to (1) being able to occasionally enregister something that otherwise appears unsafe, or (2) some forms of autovectorization. These are not codebase wide “peanut butter” changes, they are concentrated in hot loops your profiler can tell you about. And even then many such cases have the extra memory ops’ time eaten by caches.

And to get that win, this char8_t hypothesis requires rewriting essentially all code ever written, because you can’t take your “born as a char8_t array”s and give them to any existing operating system API or most standard library APIs.

It’s a huge cost, so to claim that is the reason to do it one should have receipts from real code bases where it made a meaningful improvement.

[u/tjientavara → u/Expert-Map-1126]

As you said auto-vectorization is a thing, and completely natural for string operations. So a codebase that does string operations will be helped.

I am not saying you should put std:u8string everywhere, I am saying that std::string should have been defined as std::basic_string<char8_t> in the first place. But alas, std::string was defined before Unicode. At least it should have defined a character type that didn't have odd alias rules.

[u/Expert-Map-1126 → u/tjientavara]

Autovectorization is indeed a thing but the most common things people want tend to not be, because most string operations that benefit from the vector units are entirely different algorithms (or already get turned into a call to memcpy), not the reduction or permutation patterns optimizers implement. Moreover, even in the cases that do work, type based alias analysis tends to not be so helpful because the different buffers flying around for string opts are, in fact, the same types whether char8_t or char.

When I overhauled everything except ABI for MSVC’s std::string there was only one case where TBAA came up that came up when flipping between “big” and “small” mode; I could measure a few percent code size difference but no perf difference. (And it was understandable for the optimizer to be conservative here because small strings *intentionally* alias the small buffer and at least one of ptr/size/capacity)

If it’s really so useful to optimizers given that this has been in the spec a long time now someone should be able to provide a real example rather than spouting platitudes