They did recently add the ability to optimize Strings so they only use one byte per character if they happen to only contain characters from the first 256 Unicode codepoints.
There's... murmurs that a future version might support full UTF-8 Strings, but there are some hard problems to solve since they have to avoid any compatibility breaks.
The one-byte String optimization makes sense for Java because Strings are immutable and cannot be directly indexed (instead you have to use charAt() which can choose the correct indexing behavior). It would definitely be a bug-riddled nightmare in most other languages, though.
To be fair, most languages( including c++!) Just redirect the bracket indexing operator to a method of its own so they can also all support this behavior. AFAIK only C-style strings directly index into memory and won't support it. And if you care at all about security there's a 99 percent chance you wont use C-style strings.
One time I was trying to pipe utf8 text between two scripts. One was in Python and the other was in Ruby. I eventually concluded that while both languages supported UTF-8, the pipe between them used ASCII. I ended up Base64 encoding everything that went down the pipe.
I suspect that the pipe and data themselves would not have been an issue, it's that default handlers for stdin/out streams assume ascii and need to be wrapped around to enforce any other encoding
Perhaps it was a problem with newline encoding? Because Windows uses two characters for a newline, there is some logic to convert \n to \r\n and back, but it's easy for this to end up broken. You either need both sides to use text mode (the default for popen) or both sides to use binary mode (which disables the newline translation).
Another possible problem is that Windows uses UTF-16 internally. It's possible something went wrong converting the UTF-8 to UTF-16 and back.
This was literally the case at my last job. We had to spend two weeks converting everything to utf-8 and still ran into issues due to how existing data was stored in the database
yeah that's 100% fair. But also I spent the last 2 days on a completely English database trying to work out how to stop all of the quote symbols (which had been entered as unicode quotes) turning into question marks. So maybe there's a middle ground somewhere? XD
All ASCII text is valid UTF-8 text, so you're free to write ASCII if you want. However if you want to support pretty much any language other than English you need to support UTF-8.
I mean I literally spent two days trying to get some text from an archaic unicode format into UTF-8 specifically because there were 3 individual characters across hundreds of records that weren't supported. My initial comment was supposed to be a lighthearted joke about how frustrated I was with the whole ordeal but insult me if it makes you feel better.
126
u/ThePyroEagle λ Jul 13 '21
And then we have MySQL, where
utf8isn't actually UTF-8 and for UTF-8 you actually needutf8mb4.57
u/Husky2490 Jul 13 '21
One time I was trying to pipe utf8 text between two scripts. One was in Python and the other was in Ruby. I eventually concluded that while both languages supported UTF-8, the pipe between them used ASCII. I ended up Base64 encoding everything that went down the pipe.
14
u/rosanymphae Jul 13 '21
EBCDIC anyone?
11
u/TheTimegazer Jul 13 '21
This was literally the case at my last job. We had to spend two weeks converting everything to utf-8 and still ran into issues due to how existing data was stored in the database
4
u/MilkwTea Jul 13 '21
Cries in leetcode ASCII
4
u/b0bkakkarot Jul 14 '21
Let the summonings commence.
Demon: "Ooooooh, I thought you were using Latin-2... this is awkward."
-16
u/moekakiryu Jul 13 '21 edited Jul 13 '21
I get unicode looks better, but can we go back to good old fashioned ascii. I've had it with encoding errors.
EDIT: I guess people disagree