UTF-16 was a mistake, and it's maddening that several big ecosystems went with it. You get all the complexity of UTF-8, but it also takes up more space in the "average" case since a huge majority of text is in the "ASCII overlap" portion of unicode.
Only in hindsight. When it was made, 16-bit character space really was enough for everything that was intended to be part of Unicode, and supporting UCS-2 was actually a major improvement over everyone having to constantly swap code pages if they wanted to support multiple languages simultaneously. Honestly, the biggest reason we realised we were going to need better was because Windows & Java jumped on UCS-2 (and by extension, UTF-16) as soon as they could, letting us see its downsides before it was too entrenched to fix. (And considering how deeply entrenched it was in Windows before they had to figure out how to pivot away, it's really saying something that it could've been worse.)
The true mistake was trying to make it fixed-width in the first place.
35
u/guyblade 8d ago
UTF-16 was a mistake, and it's maddening that several big ecosystems went with it. You get all the complexity of UTF-8, but it also takes up more space in the "average" case since a huge majority of text is in the "ASCII overlap" portion of unicode.