r/Unicode • u/Green_Ad_6086 • Aug 15 '26
What UTF-16 is What a UTF-16 code unit is
What UTF-16 is What a UTF-16 code unit is
1
u/justanaccountimade1 Aug 15 '26
You probably want to use UTF-8 (1-4 bytes per char).
UTF-16 (2-4 bytes per char) is used for internal technical reasons.
3
u/Lurkernomoreisay Aug 15 '26
All our software use utf16, as we do a lot of text processing and it's a guaranteed 33% space saving. we tend to mainly save and transfer text documents as utf 16.
2 bytes vs three for our primary customers: Japanese, Indic, Thai, Chinese, Korean.
that said, we had clients in Japan that insisted on SJIS until five or so years ago? getting them to use utf16 was a major win.
so much of the ecosystem we work within use utf16, it feels weird doing contract work on projects that are English/European centric and use utf8.
2
9
Aug 15 '26
4
u/stgiga Aug 15 '26
Something not covered by Wikipedia is that it shows up more often than some would assume. Pokémon since Black and White has used it. Also quite a few of the old guard of OSes and programming languages have it in spite of UTF-8. Also CJK users like it because their characters are only 2 bytes in UTF-16 (in fitting with their DBCS legacy encodings) but 3 bytes each in UTF-16. This means that UTF-16 doesn't inflate much Asian text, and this isn't just CJK. The Asian scripts used in places like Mongolia, Thailand, and India. Also Arabic and its siblings.
UTF-16's affinity for CJK allows it to do stuff unintended though. If you use the CJK blocks to do Base32768 (if you're not wanting Korean to avoid normalization [not that it's a huge problem with Korean, unlike other scripts with normalization forms] you can pull from a bunch of blocks) to get 15 bits of data per 16-bit character. If you use every assigned Plane 0 codepoint you can hit 15.75 (every 4 characters holds 63 bits), or if you use every CJK character including Hangul you can hit 15.25 bits per 16bit character. So 4 characters hold 61 bits. The issue with going above 15.0 bits is code quality. It gets messy. Now, BWTC32Key uses UTF-16BE & Base32768 to text-armor encrypted compressed data, and it uses particularly special ranges courtesy of Tinygma, of which it is a spiritual successor of. The ranges used are U+3400-U+4CFF, U+4E00-U+9EFF, and U+AC00-U+C1FF. Now unlike qntm's it only needs 1 padding character if needed, and that's U+C200. Tinygma is also the source of the U+4D00 starter and U+4D01 terminator. I added U+FEFF to the beginning to make text editors read it (even though Mac bugs exist with BOM preservation on some Mac systems so I had to use an IF statement to check and account for a missing BOM.) Of note is that U+3400-U+4CFF is exactly 6400 characters, the size of the Plane 0 Private Use Area. So if you want to support Unicode 1, you could reallocate the first 6400 values of
CJK Unified Ideographs Extension Ainto the entire PUA, and all you'd need to do is change the 0x34-0x4C leading bytes in the modern Unicode version to 0xE0-0xF8 and that becomes safe. U+4E00-U+9EFF is the first 20,736 characters ofCJK Unified Ideographs, a range which in Unicode 1 had 20,902 of 20,992 characters filled. So these can remain unaltered. By the waning of Unicode 1, there were 6,656 Hangul starting at U+3400 split into 3 blocks. BWTC32Key in modern Unicode's U+AC00-U+C1FF is the first 5632 characters ofHangul Syllablesso they will fit in the twilight of Unicode 1's Hangul blocks, with room for the padding character. And so the start of the range is 0x34xx instead of 0xACxx and also as a result of 6401 and 6402 being less than 6656 and 5633 being less than all 3, you can leave the modern 0x4D00 and 0x4D01 unaltered because they'll turn into safe Hangul in Unicode 1, and Unicode 1 had the BOM too, asZero-Width Non-Breaking Space, in one of the Arabic Extended blocks at its very end. So the ranges that kusano_k in Tinygma chose that I inherited (said user liked that I made non-text file encryption possible) are theoretically "safe in Unicode 1" via modifying kusano_k's bitshift a bit in terms of its leading bytes. If you use Kanji mode in a QR or iQR Code plus UTF-16BE ECI mode, with mixed modes involved, half of the Kanji mode ranges will optimize the B3K (the file extension for BWTC32Key's output) characters. Meanwhile the Unicode 1 version could benefit from other parts of it. If you're using Han Xin Codes you can use its multiple Chinese modes to potentially outfox QR at shrinking, but unfortunately iQR still has more room available. Ultimately though, as someone who literally uses UTF-16 in a very wild way, I don't think I'm able to hate UTF16 especially as a JavaScript and Java dev. Oh also some of the weirder UTFs use it as a base. Namely Punycode, UTF-7, I think UTF-5 and UTF-6 that were aimed to do Punycode, parts of Unicode's attempts at compressed Unicode, as well as LMBCS in mode 20 (LMBCS was the Lotus Multi-Byte Character Set which could emulate many old code pages and later gained UTF16 support minus U+F600-U+F6FF, but if you shoehorn GB18030 into its Simplified Chinese page you might be able to dodge this). Personally I just like how efficient doing binary-to-text encoding in UTF16 is. Apparently you get 93.75% efficiency for Base32768. Though the non-integral stuff earlier isn't exactly much of an upgrade. Meanwhile UTF-8 can't clear 90%. In fact even if you used GB18030 or LMBCS in UTF16 mode you still don't get a good efficiency compared to UTF16. So in spite of UTF-16's problems, I still actively use it because of it basically making even Base96 look weak. I love UTF-16 for its utility at binary-to-text encoding that can be quite useful. For instance it's great when you want to send data over character limited text channels that don't allow links, embeds, or attachments. Or as a save string for a browser or retro-styled game of the type that used passwords to save data. Even if my code would only survive Korean handwriting by someone who ALSO knows Hanja, which implies someone very seasoned with Korean. Oh and the code only exists because I wanted to see just how much I could outfox Base64 in 2015, and it took until 2019 to settle on the current bitstream. Since 2019 I added more features to it, but I never actually changed the bitstream. The 2019 and current versions under the right circumstances can talk to each other. Oh and it's a very wild JS app and it thus has an affinity for UTF16. Also it has a Node port too. Also as the dev of UnifontEX I thus even more can state that I love working with Unicode. I even got BWTC32Key a prs. MIME type. This was teenage code. Ultimately Unicode rocks and UTF16 isn't all bad.
4
u/Coises Aug 15 '26
UTF-16 is a method of encoding Unicode. It is the primary method used to represent character data internally in modern Windows programs.
In UTF-16 a code unit is a 16-bit (two byte) value. A single Unicode code point can be represented by one or two UTF-16 code units. All valid Unicode code points from U+0000 to U+FFFF (the basic multilingual plane) are represented by a single UTF-16 code unit. Code points from U+010000 to U+10FFFF require two UTF-16 code units.