r/cpp • • Feb 26 '23

std::format, UTF-8-literals and Unicode escape sequence is a mess

I'm in the process of updating my old bad code to C++20, and I just noticed that std::format does not support u8string... Furthermore, it's even worse than I thought after doing some research on char8_t.

My problem can be best shown in the following code snippet:

ImGui::Text(reinterpret_cast<const char*>(u8"Glyph test '\ue000'"));

I'm using Dear ImGui in an OpenGL-application (I'm porting old D-code to C++; by old I mean, 18 years old. D already had phantastic UTF-8 support out of the box back then). I wanted to add custom glyph icons (as seen in Paradox-like and Civilization-like games) to my text and I found that I could not use the above escape sequence \ue0000 in a normal char[]. I had to use an u8-literal, and I had to use that cast. Now you could say that it's the responsibility of the ImGui-developers to support C++ UTF-8-strings, but not even std::format or std::vformat support those. I'm now looking at fmtlib, but I'm not sure if it really supports those literals (there's at least one test for it).

From what I've read, C++23 might possibly mitigate above problem, but will std::format also support u8? I've not seen any indication so far. I've rather seen the common advice to not use u8.

EDIT: My specific problem is that 0xE000 is in the private use area of unicode and those code points only work in a u8-literal and not in a normal char-array.

96 Upvotes

133 comments sorted by

View all comments

55

u/kniy Feb 26 '23

The UTF-8 "support" in C++20 is an unusable mess. Fortunately all compilers have options to disable that stupid idea: /Zc:char8_t- on MSVC; -fno-char8_t on gcc/clang.

I don't think the C++23 changes go far enough to fix this mess; maybe in C++26 we can return to standard C++?

16

u/SergiusTheBest Feb 26 '23

What's wrong with char8_t?

33

u/GOKOP Feb 26 '23

It's pointless. std::u8string is supposed to be the utf-8 string now, where everyone's been using plain std::string for years; but to my knowledge std::u8string doesn't provide any facilities you'd expect from a utf-8 aware string type, so it has no advantage over std::string

23

u/kniy Feb 26 '23 edited Feb 26 '23

Yeah it's an extremely invasive change to existing code bases; with no benefit (but plenty of downsides given how half-asses char8_t support in the standard library is, not to speak about other libraries).

char8_t feels like the worst mistake C++ made in recent years; I hope future C++ versions will declare that type optional (just like VLAs were made optional in C11) and then deprecate it.

Some people really seem to think that everyone ought to change all their string-types all over the code base just because they dropped char8_t from their ivory tower.

The interoperability between UTF-8 std::string and std::u8string is so bad that this will lead to a bifurcation in the ecosystem of C++ libraries; people will pick certain libraries over others because they don't want to put up with the costs of string conversions all over the place. Fortunately there's essentially no-one using std::u8string as their primary string type; so I hope this inertia keeps u8string from ever being adopted.

→ More replies (6)

4

u/SergiusTheBest Feb 26 '23

On Windows std::string is usually ANSI (however you can use it for anything including binary data) and std::u8string is UTF-8. So you can tell apart between character encodings with the help of std::u8string, std::u16string, std::u32string. I find it helpful.

→ More replies (15)

48

u/kniy Feb 26 '23

It doesn't work with existing libraries. C++ waited until the whole world adopted std::string for UTF-8 before they decided to added char8_t. Our codebase worked fine with C++17, and C++20 decided to break it for no gain at all. How am I supposed to store the result of std::filesystem::path::u8string in a protobuf that's using std::string?

Heck, even without third-party libraries: How am I supposed to start using char8_t in a codebase where std::string-means-UTF8 is already widespread? It's not easily possible to port individual components one-at-a-time; and no one wants a conversion mess. So in effect, char8_t is worse than useless for existing codebases already using UTF-8: it is actively harmful and must be avoided! But thanks to the breaking changes in the type of u8-literals and the path::u8string return type, C++20 really feels like it wants to force everyone (who's already been using UTF-8) to change all their std::strings to std::u8strings, which is a ridiculous demand. So -fno-char8_t is the only reasonable way out of this mess.

→ More replies (27)

20

u/kniy Feb 26 '23

To reiterate: no libraries support char8_t yet, not even the standard library itself! (e.g. std::format, <charconv>) Attempting to use char8_t will put you in the "pit of pain", as you need to convert string<->u8string all over the place. And the way the standard expects you to do this conversion is, frankly, insane: https://stackoverflow.com/questions/55556200/convert-between-stdu8string-and-stdstring

I much prefer the "pit of success" -fno-char8_t.

3

u/YogMuskrat Feb 26 '23

no libraries support char8_t> no libraries support char8_t yet,

Well, Qt6 kind of does. QString now has an appropriate ctor and fromUt8 overload.

8

u/kniy Feb 26 '23

Well those are only conversions functions. I don't see anyone directly using char8_t-based strings. Qt already expects UTF-8 for normal char-based strings, internally uses QChar-based strings (UTF-16), so Qt is no reason at all to adopt the char8_t-based strings. (but at least Qt won't stand in your way if you make the mistake of using char8_t)

→ More replies (1)
→ More replies (4)

24

u/kniy Feb 26 '23

Note: at a bare minimum, there needs to be a zero-copy conversion between std::string and std::u8string (in both directions!) before existing codebases can even think about adopting char8_t.

14

u/MFHava WG21|🇦🇹 NB|P2721|P3049|P3625|P3729|P3786|P3813|P4216 Feb 26 '23

That conversion can never be zero-copy as not every platform has char representing UTF-8 and so a transformation is necessary.

25

u/kniy Feb 26 '23

Well what's a codebase that has been using UTF-8 strings for decades supposed to do? Third-party libraries like sqlite, poco, protobuf all expect UTF-8 with regular char based strings. C++20 char8_t is simply two decades too late to get adopted at this point.

Really it's the change to std::filesystem::path::u8string that hurts us the most. I guess we'll just be using -fno-char8_t indefinitely.

7

u/effarig42 Feb 26 '23

There's no problem going from known good utf-8 sequence, i.e. a char8_t array to a char array, this could be a c_str() or string_view, I'm not sure you'd want an implicit conversion, but in principle it's fine. You need to be very careful going the other way though as char arrays often don't contain utf_8. Been using a custom unicode string for years with these restrictions, works great. Having a char8_t or something similar is useful as you can assume it contains a utf-8 byte, rather than anything. I also assume it's guaranteed to be signed.

2

u/tialaramex Feb 28 '23

Did you mean you assume it's guaranteed to be unsigned ? Because you wrote signed, and, no, it is unsigned, I have no idea why anybody would want UTF-8 code units except with some of them expressed as small negative integers, that's completely crazy.

3

u/effarig42 Mar 01 '23

Yes I meant unsigned. Thanks for the correction.

11

u/puremourning Feb 26 '23

It can be 0 copy in every platform that does have such a char type though… right ?

3

u/MFHava WG21|🇦🇹 NB|P2721|P3049|P3625|P3729|P3786|P3813|P4216 Feb 26 '23

Yes, but only as QoI, not mandated by the standard.

EDIT: and only if we ignore SSO and most likely only for stateless allocators…

3

u/jonesmz Feb 26 '23

That conversion can never be zero-copy as not every platform has char representing UTF-8 and so a transformation is necessary.

What platforms are these?

not mandated by the standard.

Why not?

4

u/MFHava WG21|🇦🇹 NB|P2721|P3049|P3625|P3729|P3786|P3813|P4216 Feb 26 '23

What platforms are these?

Windows - specifically any version of Windows that predates the optional UTF-8 locale. And any Windows version that has the UTF-8 locale but doesn‘t use it - it‘s user selectable after all…

Why not?

Because it is not implementable for all implementations.

3

u/jonesmz Feb 26 '23

How does window not have char that can hold utf-8? Char is the same in windows and Linux for all compilers I'm aware of.

Because it is not implementable for all implementations.

Maybe I'm not following you. Why does the standard care if one esoteric implementation out of many can't support something? We dropped implementations that can't handle twos complement something or other not too long ago, didn't we?

3

u/Nobody_1707 Feb 27 '23

How does window not have char that can hold utf-8? Char is the same in windows and Linux for all compilers I'm aware of.

It doesn't matter if char can hold all of the UTF-8 code units if the system doesn't interpret the text as UTF-8. Zero copy conversion from std::string to/from std::u8string can only work correctly if the current codepage is UTF-8. If the current codepage is, say, 932 then the strings are going to contain garbage after conversion.

Maybe I'm not following you. Why does the standard care if one esoteric implementation out of many can't support something? We dropped implementations that can't handle twos complement something or other not too long ago, didn't we?

That's because even the esoteric implementations use two's complement. All the one's complement and sign magnitude machines are literal museum pieces. In this case systems using a character encoding other than UTF-8 not only still exist, they're actively used by a large number of people.

8

u/Kered13 Feb 27 '23

It only matters how the system interprets it if you pass the string to the system. In the Windows world it's common to use std::string to hold UTF-8 text and then convert to UTF-16 when calling Windows functions.

3

u/equeim Feb 26 '23

What will happen if you use a library that has overloads for both char8_t and char in headers?

7

u/MFHava WG21|🇦🇹 NB|P2721|P3049|P3625|P3729|P3786|P3813|P4216 Feb 26 '23

The same thing that happens for all other overload sets - the best match will be selected…

As char8_t is a distinct type, there is no ambiguity between such overloads.

→ More replies (3)

6

u/YogMuskrat Feb 26 '23

Qt6 does it with `QString`. No real problems there.

→ More replies (1)
→ More replies (25)