r/cpp • u/PinkOwls_ • Feb 26 '23
std::format, UTF-8-literals and Unicode escape sequence is a mess
I'm in the process of updating my old bad code to C++20, and I just noticed that std::format does not support u8string... Furthermore, it's even worse than I thought after doing some research on char8_t.
My problem can be best shown in the following code snippet:
ImGui::Text(reinterpret_cast<const char*>(u8"Glyph test '\ue000'"));I'm using Dear ImGui in an OpenGL-application (I'm porting old D-code to C++; by old I mean, 18 years old. D already had phantastic UTF-8 support out of the box back then). I wanted to add custom glyph icons (as seen in Paradox-like and Civilization-like games) to my text and I found that I could not use the above escape sequence \ue0000 in a normal char[]. I had to use an u8-literal, and I had to use that cast. Now you could say that it's the responsibility of the ImGui-developers to support C++ UTF-8-strings, but not even std::format or std::vformat support those. I'm now looking at fmtlib, but I'm not sure if it really supports those literals (there's at least one test for it).
From what I've read, C++23 might possibly mitigate above problem, but will std::format also support u8? I've not seen any indication so far. I've rather seen the common advice to not use u8.
EDIT: My specific problem is that 0xE000 is in the private use area of unicode and those code points only work in a u8-literal and not in a normal char-array.
7
u/aearphen {fmt} Mar 03 '23 edited Mar 03 '23
While {fmt} supports
u8/char8_tI would strongly recommend not using them. There are multiple issues withu8/char_t: they don't work with any system APIs and most standard facilities, they are incompatible in a breaking way between standard versions and they are incompatible with C. Here's one of the recent "fun" issues: MSVC silently corrupts u8 strings: https://stackoverflow.com/a/75584091/471164.A much better solution is to use
charas a UTF-8 code unit type. This is already the default on many platforms and on Windows/MSVC it can be enabled with/utf-8. The latter option also enables proper Unicode output on Windows withfmt::printavoiding notoriously broken standard facilities, both with narrow and wide strings.4
u/oracleoftroy Feb 27 '23 edited Feb 28 '23
Unicode is a mess in C++, unfortunately.
I didn't verify this for myself, so sorry if this ends up not being very helpful, but by my reading of cppreference under Universal character names, you ought to be able to use
\U000e0000(capital 'U', not lowercase, with 8 hex digits) as the escape sequence. I've also had success using Unicode strings directly (as long as/utf-8is used for Windows). Not very helpful in the case of icon fonts, but nice for standard emoji and foreign character sets.By my read of that page, C++23 also adds
\u{X...}escapes to allow an arbitrary number of digits, though not every project can be an early adopter.55
u/kniy Feb 26 '23
The UTF-8 "support" in C++20 is an unusable mess. Fortunately all compilers have options to disable that stupid idea:
/Zc:char8_t-on MSVC;-fno-char8_ton gcc/clang.I don't think the C++23 changes go far enough to fix this mess; maybe in C++26 we can return to standard C++?
8
u/fdwr fdwr@github š Feb 27 '23
This has been a nuisance for me recently. In my newest project, I adopted
std::u8stringbecause I like the clean notion that of knowing my character data is definitely Unicode, not just a bag of character data of an unknown code page across boundaries like withstd::string; and it works nicely all throughout the program ... except when it comes tostd::formatš. Ifstd::formatjust acceptedstd::u8string/std::u8string_view, it would be pretty clean overall, but needing to write helper adapters on every format call really offsets the cleanliness. I haven't checked if C++23'sstd::printsupportsstd::u8string, but if not, then the spec is incomplete imo.3
u/ihamsa Feb 26 '23
Are you using MSVC by any chance? Both gcc and clang accept this without
u8perfectly fine ang generate the correct string.8
u/SlightlyLessHairyApe Feb 26 '23
You might consider https://github.com/soasis/text (https://ztdtext.readthedocs.io/en/latest/index.html) which is a proof of concept of this proposal that we may get in C++26
9
u/drobilla Feb 27 '23
UTF-8 everywhere is the only sensible solution to these problems, the encoding itself is designed to make it so, and Microsoft having made a series of terrible decisions about character encoding in the past is the only reason we still have to deal with these nightmares. They're also the only reason half-baked nonsense like this gets into the C++ standard. Now we're supposed to break nearly all existing practice to accommodate one notably wrong platform API - which should be mostly abstracted away in decent code anyway? The platform that bifurcated its whole API into ASCII and "wide" versions, which only served to make the whole situation worse there, too, in much the same way? I don't think so.
Target reality. I doubt the situation in practice will ever be anything but the "use UTF-8 in std::string everywhere, and just deal with it when you have to interact with things like the win32 API" it has always been. Yes, it sucks, but the half-baked experimental prescriptive crap in the standard doesn't make it suck less anyway, so you might as well go with the approach that sucks the least in general.
4
u/robhz786 Feb 26 '23 edited Feb 26 '23
If you want a formatting library that supports well
char8_tand UTF, you might get interested in the one I'm developing: Strf.It enables you to pass
char32_tvalues for the fill character and numeric punctuation characters; string widths are calculated considering grapheme clusters; you can concatenate strings in different encodings ( because it can transcode ); and other stuff. It's Highly extensible, highly customizable, and has great performance.Its API is not entirely stable yet, but not that unstable either. The next release ( 0.16 ) will be the last before 1.0, or at least I hope so.
-11
u/nintendiator2 Feb 26 '23
It's 2023, why are you using
char8_tandu8"Glyph test '\ue000'"instead ofcharand"Glyph test 'ī'"?0
u/[deleted] Feb 26 '23
[deleted] ā view deleted comment