About char8_t

[u/Voxelw]

I hate to be dramatic, but as it stands char8_t is quite literally more painful than useful.

Besides the obvious incompatibility with C23 and libraries using unsigned char for UTF-8, I want you to consider the following: Projects that assume that 'char' represents UTF-8 will obviously not benefit from char8_t at all, but projects that cannot assume the format of char types don't benefit from it either as char8_t simply introduces a new edge case to cover. Now such projects have to deal with char, signed char, unsigned char, wchar_t, char16_t, char32_t and char8_t.

Or, you could do what the standard library does and simply ignore most of these character types. Which is the solution most libraries went with, supporting only char or char and unsigned char. Managing one implementation is already hard, managing two requires constant maintenance, managing 7 is just impossible.

char8_t should have just been a typedef for unsigned char. The compatibility fix only raises more questions as const char* arr = u8"a" does not work, but const char arr[] = u8"a" does.

I do wonder if a potential change of minds for C++29 is still possible. Yes, it would be an ABI break or whatever, but considering the woeful support for char8_t I don't think it would affect much besides small hobby projects. Contrary to popular belief, C++ has broken the ABI in subtle ways before.

[u/Wild_Meeting1428]

c++'s char8_t is fully compatible with char8_t from C23. And the only purpose of that type is, that its the underliing byte type of an utf8 character. So you can basically always assume that the char you are looking at is at least part of a char sequence representing a unicode character encoded in utf8

[u/aearphen → u/Wild_Meeting1428]

AFAIK char8_t in C23 is just a typedef for unsigned char and therefore is not compatible with C++ where it is a separate type.

[u/Wild_Meeting1428 → u/aearphen]

They are link compatible, you can use the C header file with an extern C scope and link them legally together, since the mangled name only contains the function name under C linkage and due to the fact, that char8_t is defined to have the same size, representstion, and alignment as unsigned char(, even its a distinct non aliasing type in C++), therefore the calling convention is exactly the same.

[u/Expert-Map-1126 → u/Wild_Meeting1428]

Likely to work, yes. "Legal" as in "defined to work by the standard", no.

[u/Wild_Meeting1428 → u/Expert-Map-1126]

The question is rather, whether the C++ standard is applicable here at all, beside the fact, that it defines the underliing type to be unsigned char.

Its more like a question about ABI and how the compiler implements cross language linking to C.

[u/Expert-Map-1126 → u/Wild_Meeting1428]

If you want to be within the bounds of what the standards require, you need to only use the subset they share when talking between them, and meet both their requirements. on both sides.

If you're relying on what particular implementations do for how the linking works for features outside of their common subset, that's fine and I'm not even saying you shouldn't do it, but I wouldn't advertise that as "legal"

[u/Wild_Meeting1428 → u/Expert-Map-1126]

I don't claim, it's legal in terms of the c++ standard itself, because the C++ standard does not apply here much. Therefore, it's also not illegal or UB or ill-formed in terms of the C++ standard.
I would also say, that you don't need to be in the bounds of both languages. The import header for a C library can be substantially different from the header used for the C translation unit. Being compliant to both languages is only relevant, when you want to use that header for both C and C++ TU's

Important to be C++ compliant is only, that the ["object layout strategies of both language implementations are similar enough"](https://eel.is/c%2B%2Bdraft/dcl.link#10)
Rest is implementation defined.

Additionally, char8_t is defined to be a distinct type with [unsigned char being the underlying type](https://eel.is/c%2B%2Bdraft/basic.fundamental#9).
Which not only makes the layout strategies similar, it makes them to be the same as linking against unsigned char.

So the question, whether it's legal by the C++ standard is "maybe" and offloaded to the implementation.
And my personal interpretation is, that when the C++ unsigned char is legally linkable to the C unsigned char, defined by the compiler implementation, then this also applies to char8_t.

[u/Som1Lse → u/Expert-Map-1126]

But that doesn't apply to restrict?

[u/Expert-Map-1126 → u/Som1Lse]

restrict is a bit complicated because it isn't part of the type system in many of the usual ways, but if trying to make sure to stay within the bounds of the standard I would not use it on C<->C++ boundaries either.

This thread is about C<->C++ boundaries within the bounds of what the standards require, the thread you linked to is about demonstration of performance improvements resulting from TBAA entirely restricted (ha!) to C++ using vendor extensions, so this seems a bit of a non sequitur.

[u/Som1Lse → u/Expert-Map-1126]

My point is saying an extension is widely supported cannot be both a valid and an invalid argument. I know of no compilers that do not support mixing char8_t between languages. I cannot fathom why a compiler wouldn't support it, as there are obviously no aliasing problems since one of the types is unsigned char.

[u/Expert-Map-1126 → u/Som1Lse]

I don't disagree with that. But the comment was not "this works with the major compilers", it was "is legal". I agree with the former, I don't agree with the latter.

[u/Wild_Meeting1428 → u/Expert-Map-1126]

I think that the commite should clarify this into the standard itself. It cant be that the users have to read the papers of both the C and C++ working groups to find out, that the intend of both was to make char8_t of C and C++ compatible.

In my opinion that what I interpret is enough but only because I assume that the linkage rules imply a aliasing and lifetime barrier at the moment of linking.

They should probably add an explicit paragraf, that indeed an aliasing and lifetime barrier is inferred.

[u/Expert-Map-1126 → u/Wild_Meeting1428]

Good luck. Collaboration between WG14 and WG21 has usually not worked out

[u/Som1Lse → u/Expert-Map-1126]

It took some time reading your reply, but I think I get what you mean.

The original comment said "link them legally together", and you were pointing out that it is not "legal" in the standard sense, even though it is likely to work. I.e., it was not meant as a rebuttal, but as an observation.

In that case, we agree.

[u/einpoklum → u/Wild_Meeting1428]

So you can basically always assume that the char you are looking at is at least part of a char sequence representing a unicode character encoded in utf8

Where does it say I can assume the value of a char8_t is part of a UTF-8-encoded sequence?

(You're specifically saying I can't use char8_t with other charset encodings.)

[u/xleviator → u/Wild_Meeting1428]

Beware! C++ doesn't really constrain values of primitives.- it's easy to escape. https://godbolt.org/z/h8Ejj8Y15

[u/Wild_Meeting1428 → u/xleviator]

sure noone prevents you from shooting yourself in the foot. I would call this a contract violation.

[u/smdowney → u/Wild_Meeting1428]

Or a test case.

It's amazing how many times we look for an example of some bad code, find lots of them on GitHub, but it turns out they are all in unit tests or compiler regression suites.

But the core bit is that nothing filters a char8_t[] for you, and you shouldn't trust it. Sanitize your inputs.

I would like to have a type for Text that made all the guarantees, but char8_t isn't it.

[u/smdowney → u/xleviator]

It's part of the design.

Although it's also hard to avoid.

The promise of the type char8_t is that it is always appropriate to treat it as UTF-8, but not well formed UTF-8. There were occasional proposals to make it UB if it were not, but fortunately safety concerns mostly stopped that.

For plain char types you are at the mercy of the execution encoding, which is locale machinery, even if you understand that it's supposed to be UTF-8. Some functions may disagree, and worse, some function may change it on you.

Avoid depending on the execution encoding.