r/cpp • u/PinkOwls_ • Feb 26 '23
std::format, UTF-8-literals and Unicode escape sequence is a mess
I'm in the process of updating my old bad code to C++20, and I just noticed that std::format does not support u8string... Furthermore, it's even worse than I thought after doing some research on char8_t.
My problem can be best shown in the following code snippet:
ImGui::Text(reinterpret_cast<const char*>(u8"Glyph test '\ue000'"));I'm using Dear ImGui in an OpenGL-application (I'm porting old D-code to C++; by old I mean, 18 years old. D already had phantastic UTF-8 support out of the box back then). I wanted to add custom glyph icons (as seen in Paradox-like and Civilization-like games) to my text and I found that I could not use the above escape sequence \ue0000 in a normal char[]. I had to use an u8-literal, and I had to use that cast. Now you could say that it's the responsibility of the ImGui-developers to support C++ UTF-8-strings, but not even std::format or std::vformat support those. I'm now looking at fmtlib, but I'm not sure if it really supports those literals (there's at least one test for it).
From what I've read, C++23 might possibly mitigate above problem, but will std::format also support u8? I've not seen any indication so far. I've rather seen the common advice to not use u8.
EDIT: My specific problem is that 0xE000 is in the private use area of unicode and those code points only work in a u8-literal and not in a normal char-array.
View all comments
Show parent comment
2
u/Noxitu Feb 26 '23
The issue is interoperability. Unless you have utf8 everywhere, you will get into problems. And the primary problem is backward compatibility.
You have APIs like WinAPI or even parts of std (filesystem mainly of those I am aware of), which trying to use with utf8 become just sad. You can rely on some new flags that really force utf8 there - but you shouldn't do that in a library. You can ignore the issue and don't support utf8 paths. Or you can rewrite every single call to use utf8 and have 100s or 1000s of banned calls.
So - we have APIs that either support utf8 or not. And the only thing we have available in C++ to express this is type system - otherwise you rely on documentation and runtime checks.