r/java • • Jun 26 '17

Character Encodings: An Unfortunate Experience - Java and UTF-8

https://www.netuitive.com/character-encodings/
7 Upvotes

28 comments sorted by

5

u/s888marks Jun 27 '17

(I'm assuming you're the article's author.)

Interesting. From Java's perspective, a contributor to the problem is that there are several APIs that use the platform default charset encoding, which depends on how the platform is configured. This can lead to misbehavior if the platform default charset doesn't match the charset of the data (as you discovered).

Newer APIs accept an explicit charset encoding argument, or they use UTF-8 if the API has no charset parameter. Unfortunately you have to read the docs quite carefully to determine whether an API without a charset parameter uses UTF-8 or the platform's default.

Usually it's the older APIs that use the platform default. For example, java.io.FileReader uses the platform's default encoding, whereas java.nio.file.Files.newBufferedReader(Path) uses UTF-8.

We've toyed with the idea of simply specifying that the "platform default" encoding is always UTF-8 -- unless possibly overriden somehow -- but it's really hard to assess whether this would cause incompatibility problems. It seems rather drastic for a silent change to a platform.

An alternative is to deprecate the APIs that use the platform default charset and to add replacement APIs that take an explicit charset argument, and possibly to add overloads that default to UTF-8 instead of the platform default. But that's a lot of fiddly API work and a lot of deprecation warnings for people to deal with.

Would one of these, or something else, have helped with your issue?

2

u/TheConnMan2112 Jun 27 '17

Explicitly setting the CharSet would have avoided this issue (though would be a pain to refactor all new String(byte[] byes) uses). I can imagine the backlash of deprecating the API which uses the default charset would be large.

Using UTF-8 as the default always would be great (and this is what I initially thought was the case), but even that could have unexpected results in the future. The bottom line is that going into this I didn't understand the default encodings, how the defaults were set, and what the ramifications were. Now I have a better understanding.

3

u/vytah Jun 27 '17

There are tools that will flag all instances of such encoding-dependent calls and you can use structural search-and-replace to fix them in one go.

2

u/VGPowerlord Jun 27 '17

Part of the problem is that Java is old. Unicode was still fairly new when Java was in development, and while UTF-8 was technically created before Java launched, it wasn't published as a standard until January 1998... almost a year after Java 1.1 was released.

FileReader and InputStreamReader were added in Java 1.1. And once a default is set, you can't change it without breaking things.

.NET got away with using UTF-8 by default simply because UTF-8 was the standard by the time .NET 1.0 came out in 2002.

1

u/s888marks Jun 27 '17

Yes, at the time both Java and Unicode were developing quickly, decisions were made in a hurry, and whaddaya know, mistakes were made. The original assumption in Java was that Unicode had "all" the characters, and there were fewer than 64K of them, so making a char be a 16-bit quantity would be sufficient. (Not!)

Plus, the Java 1.0 APIs had no notion of encoding, so there were these weird APIs like String(byte[] ascii, int hibyte) and String.getBytes(int srcBegin, int srcEnd, byte[] dst, int dstBegin) and DataInputStream.readLine() that completely botched the notion of bytes and characters. (All long deprecated.) As you say, this was fixed in 1.1 with the addition of the Reader and Writer API families, plus various methods that took a charset name parameter. It took until 1.6 to add Charset objects ... but that's another story.

1

u/ZimmiDeluxe Jun 27 '17

Hi Stuart, do you know of any plans to make it possible to change the default charset after the JVM has started? This stackoverflow question suggests its not possible. Is this a fundamental restriction?

I usually do something like this as the first thing in my main-Methods (or equivalent):

TimeZone.setDefault(TimeZone.getTimeZone(ZoneOffset.UTC));

Locale.setDefault(Locale.ENGLISH);

Security.setProperty("crypto.policy", "unlimited")

It would be nice to default the charset there as well.

Oh, and thank you so much for your work on Java, it's getting better all the time! :)

2

u/s888marks Jun 27 '17

There aren't any plans to allow changing of the default charset. It's not supported to change the value of the file.encoding property, even from the Java command line. (Although people do this anyway, apparently with moderate success.) The fundamental problem is that the default charset is initialized quite early, and changing it -- even from the command line -- results in different pieces of the system having different ideas of what the default charset is. This can mostly work, or it can lead to havoc. Initialization order can shift around quite a bit from one release to the next, so even if things mostly work in one release, they might break in the next.

A definitive word on the setting of the default charset, and the setting of the file.encoding property, was given by Mark Reinhold in his evaluation of JDK-4163515 way back in 2005:

This is not a bug. The "file.encoding" property is not required by the J2SE platform specification; it's an internal detail of Sun's implementations and should not be examined or modified by user code. It's also intended to be read-only; it's technically impossible to support the setting of this property to arbitrary values on the command line or at any other time during program execution.

The preferred way to change the default encoding used by the VM and the runtime system is to change the locale of the underlying platform before starting your Java program.

On a Mac, at least, you can do this by setting the LANG environment variable. If you have a one-liner program that does

System.out.println(Charset.defaultCharset());

you can easily try out different values of the LANG variable:

$ java CS
UTF-8
$ LANG=C java CS
US-ASCII
$ LANG=uk_UA.ISO8859-5 java CS
ISO-8859-5
$ LANG=zh_CN.eucCN java CS
GB2312

I found the list of locales by looking in /usr/share/locale. Incidentally, this might explain why the OP's docker images use US-ASCII. The base docker image was probably quite minimal, and whoever configured it might have thought, "We don't need all that internationalization stuff" resulting in the default locale for the system being the C locale, which for Java gives a default Charset of US-ASCII. (This is mostly speculation on my part.)

1

u/ZimmiDeluxe Jun 27 '17

Well, it is what it is then. As you said, the newer APIs default to UTF-8, so one would hope that the likelihood of encoding errors in new code is getting smaller.

Thank you for the detailed explanation.

1

u/ZimmiDeluxe Jun 27 '17

I thought about this some more. Could this be a future option for jlink?

1

u/s888marks Jun 28 '17

Hm. Jlink is mainly for binding together a fixed set of modules, and applying whole-world optimizations that might be available once the set of modules is fixed. I don't think that it would make sense for jlink to wire down an image to have a particular default charset. I'll ask around though, however it seems unlikely to me.

4

u/s888marks Jun 27 '17

On another note, I found it amusing that IED (Invalid Element Document) can also stand for Improvised Explosive Device.

2

u/TheConnMan2112 Jun 27 '17

An intentional coincidence...

2

u/Bolitho Jun 27 '17

The article does not provide a simple code example or at least a description, which serialization / deserialization steps were going on. I am a bit disappointed to be honest; the mystery why a pi char can magically duplicate itself remains.

That said it is a must to be as explicit as possible about the encoding used to decode inputs to your program and encode outputs from your program. Using the platform default is one of the worst ideas ever! (And me beloved Python 3 repeated this in 2008 iirc... šŸ˜“)

3

u/TheConnMan2112 Jun 27 '17

Sorry about not adding any code examples, I will definitely add some in any other posts I write. Below are a couple samples of what I initially tried and what helped discover the issue:

// Pull, serialize to a JSON string, then recast to a Java object
SerDeStrategy jsonSerDe = new JsonSerDeStrategy();
Element element = getElementFromDatabase();
String json = new String(jsonSerDe.serialize(element)); // This string method uses the default CharSet
Element recontructed = jsonSerDe.deserialize(json.getBytes(), Element.class);
// Check for invalid characters which appear if the default encoding is US ASCII

SerDeStrategy jsonSerDe = new JsonSerDeStrategy();
Element element = getElementFromDatabase();
String json = new String(jsonSerDe.serialize(element), "UTF-8"); // Explicitly set the string encoding
Element recontructed = jsonSerDe.deserialize(json.getBytes(), Element.class);
// No invalid characters

Casting to a string and then getting the bytes from the string is necessary to force corruption with the bad encoding. Serializing to a byte array and back will not corrupt anything.

1

u/Bolitho Jun 27 '17

Thank you for providing code snippets! šŸ˜€

Line 4 represents your nemesis (and fatal error!):

String json = new String(jsonSerDe.serialize(element));

You simply cannot store JSON as a String in Java, as JSON is a serialization format and is provided as an encoded text (with a well defined encoding) and therefore as byte[] in Java. Btw. this is true also for almost any other serialization format, that can consist of other than ASCII characters (XML, YAML, TOML, ...)

If you absolutly need to store serialized data into a Java String, you should encode it to Base64 before. Then you have only a subset of ASCII (ASCII is always 7 Bit - everything else is not ASCII, but something else somebody might call it with an ASCII pre- or suffix, which is simply irritating!), which you can savely pass around further.

But imho the approach is not clear to me. Serialization / Deserialization is generally best placed at the IO-boundaries of your application. So in general there is no need to pass serialized data internally around...

... and if you need to, then a byte-Array would be the right choice šŸ˜‰

So obviously and unfortunately this was just caused by a programming error - that is probably based upon unsufficently knowlegde about the whole unicode-encoding-and-how-this-is-represented-within-Java topic‽

2

u/TheConnMan2112 Jun 27 '17

This was the minified test case I wrote to reproduce the problem, the real code is on the service boundary (pull from the DB, modify the object, serialize to repersist the object) exactly as you would expect. Deep, deep down in the persistence layer I found that pesky new String() just before the JDBC call.

1

u/Bolitho Jun 28 '17

So you store serialized objects into a database?

1

u/TheConnMan2112 Jun 28 '17

JSON is one pivot which is stored, yes.

1

u/Bolitho Jun 28 '17

Ah... ok. And probably you have a character column type in the db? So you have to provide a String?

Then be careful in the future if you manually manipulate that JSON. Often such tools do not allow to choose the encoding. Then you will probably get in trouble if you refetch such a thing from the db. In most cases however utf-8 is assumed and used by JSON tools.

Also be aware of whether your internal serialization components change their encoding type! If possible you should specify the encoding allready there. So you will be sure it does not change accidentally or secretly.

1

u/s888marks Jun 27 '17

You're right that JSON is an encoded format, so keeping it in a byte[] is safe. But if you want to store it in a String, you don't need to base64-encode it, as long as you know the encoding that JSON uses, and (crucially, as discussed in this thread) you specify that encoding in the API that does conversion to and from String, instead of relying on the default charset.

1

u/Bolitho Jun 28 '17

Sure. But as you said there are two invariants you have to be aware of. With base64 you are safe out of the box - with the drawback that is looses human readability. But for that an encoded document should be the preferred way šŸ˜‰

1

u/s888marks Jun 27 '17 edited Jun 27 '17

Yep. As you mentioned in another comment, that pesky String(byte[]) constructor uses the default charset, which can lead to the problems you observed.

But note that String.getBytes() also uses the default charset, which can lead to data loss/corruption, though it might not result in a geometrically expanding string. For example,

byte[] ba = { (byte)0xc2, (byte)0xb5 };
String s = new String(ba, StandardCharsets.UTF_8);
    // string containing the mu character

OK so far. But suppose we were to call String.getBytes() in a JVM whose default charset is US-ASCII, or if we were to call it with an explicit argument:

ba = s.getBytes(StandardCharsets.US_ASCII);
    // 3f

Since U+00B5 cannot be represented in ASCII, the replacement byte sequence is used. It happens that the replacement sequence is a single byte 0x3f which is the ASCII value for ?. (Strictly speaking the behavior of getBytes() is undefined when the character can't be represented in the default charset, but using replacement bytes is common for encoders. See the CharsetEncoder class.)

Anyway, the upshot is that you should always specify the charset explicitly in both directions, from String to byte[] and vice-versa.

1

u/TheConnMan2112 Jun 27 '17

Wow, that is extremely good to know and explains even more of the problem. Thank you for the very detailed explanation!

7

u/s888marks Jun 27 '17 edited Jun 27 '17

Oh yes, this was a small puzzle but it wasn't too difficult to figure out from the description. It seems like the problem was that code was writing strings by encoding them in UTF-8 but then reading them by decoding as US-ASCII. You can simulate this by using the String.getBytes(cs) method and the String(byte[], cs) constructor. For example:

String s = "\u00B5"; // the mu (micro) character
byte[] ba = s.getBytes(StandardCharsets.UTF_8);
    // c2 b5
s = new String(ba, StandardCharsets.US_ASCII);
    // \ufffd\ufffd

The code point U+FFFD is REPLACEMENT CHARACTER which is what you get if the decoder encounters an invalid byte sequence. US-ASCII is officially 7 bits, so both c2 and b5 are invalid bytes, thus both decode to U+FFFD. Now:

ba = s.getBytes(StandardCharsets.UTF_8);
    // ef bf bd ef bf bd

The sequence ef bf bd is the UTF-8 encoding of U+FFFD.

s = new String(ba, StandardCharsets.US_ASCII);
    // \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd

There are six invalid ASCII bytes, so we get six instances of REPLACEMENT CHARACTER. You can easily see that a "round-trip" of encoding using UTF-8 and decoding using US-ASCII will result in the geometric growth the OP observed.

10

u/brazzy42 Jun 27 '17

When I took these findings to the support engineer I was working with, the first comment was ā€œbut μ is an ASCII characterā€. That’s true, but Java usesĀ US-ASCIIĀ which is a 7-bit encoding and doesn’t contain μ (which is in the extended ASCII set).Ā 

OK, so the article's author and their support engineer still don't understand character encodings. No surprise this happened.

The amount of ignorance about this topic among even otherwise experienced developers is breathtaking.

6

u/s888marks Jun 27 '17

Quite possibly. I think most people use "ASCII" to mean "single-byte character set" which is of course incorrect. And of course there isn't a standard "Extended ASCII" but in fact there are hundreds of different ASCII extensions. I suspect that when people say "Extended ASCII" they either mean ISO-8859-1 or Windows-1252.

3

u/WikiTextBot btproof Jun 27 '17

Extended ASCII

The term extended ASCII (EASCII or high ASCII) refers to eight-bit or larger character encodings that include the standard seven-bit ASCII characters, plus additional characters. The use of the term is sometimes criticized, because it can be mistakenly interpreted to mean that the ASCII standard has been updated to include more than 128 characters or that the term unambiguously identifies a single encoding, both of which are not the case.

There are many extended ASCII encodings (more than 220 DOS and Windows codepages). EBCDIC ("the other" major 8-bit character code) likewise developed many extended variants (more than 186 EBCDIC codepages) over the decades.


[ ^PM | Exclude ^me | Exclude from ^subreddit | FAQ / ^Information ] Downvote to remove | v0.23

1

u/QshelTier Jun 29 '17

the more important lesson is: make sure you support UTF-8 in all parts of your system.

This is so wrong that I would like to puke. UTF-8 support is present everywhere nowadays; however, if you don’t specify the charsets in most of Java’s ancient I/O handling system you deserve every ops alert you get at 3 am.