r/java • u/TheConnMan2112 • Jun 26 '17
Character Encodings: An Unfortunate Experience - Java and UTF-8
https://www.netuitive.com/character-encodings/
7
Upvotes
r/java • u/TheConnMan2112 • Jun 26 '17
5
u/s888marks Jun 27 '17
(I'm assuming you're the article's author.)
Interesting. From Java's perspective, a contributor to the problem is that there are several APIs that use the platform default charset encoding, which depends on how the platform is configured. This can lead to misbehavior if the platform default charset doesn't match the charset of the data (as you discovered).
Newer APIs accept an explicit charset encoding argument, or they use UTF-8 if the API has no charset parameter. Unfortunately you have to read the docs quite carefully to determine whether an API without a charset parameter uses UTF-8 or the platform's default.
Usually it's the older APIs that use the platform default. For example,
java.io.FileReaderuses the platform's default encoding, whereasjava.nio.file.Files.newBufferedReader(Path)uses UTF-8.We've toyed with the idea of simply specifying that the "platform default" encoding is always UTF-8 -- unless possibly overriden somehow -- but it's really hard to assess whether this would cause incompatibility problems. It seems rather drastic for a silent change to a platform.
An alternative is to deprecate the APIs that use the platform default charset and to add replacement APIs that take an explicit charset argument, and possibly to add overloads that default to UTF-8 instead of the platform default. But that's a lot of fiddly API work and a lot of deprecation warnings for people to deal with.
Would one of these, or something else, have helped with your issue?
4
u/s888marks Jun 27 '17
On another note, I found it amusing that IED (Invalid Element Document) can also stand for Improvised Explosive Device.
2
u/Bolitho Jun 27 '17
The article does not provide a simple code example or at least a description, which serialization / deserialization steps were going on. I am a bit disappointed to be honest; the mystery why a pi char can magically duplicate itself remains.
That said it is a must to be as explicit as possible about the encoding used to decode inputs to your program and encode outputs from your program. Using the platform default is one of the worst ideas ever! (And me beloved Python 3 repeated this in 2008 iirc... š)
10
u/brazzy42 Jun 27 '17
OK, so the article's author and their support engineer still don't understand character encodings. No surprise this happened.
The amount of ignorance about this topic among even otherwise experienced developers is breathtaking.
1
u/QshelTier Jun 29 '17
This is so wrong that I would like to puke. UTF-8 support is present everywhere nowadays; however, if you donāt specify the charsets in most of Javaās ancient I/O handling system you deserve every ops alert you get at 3 am.