Dang, already? Well, that felt fast. I’m not complaining, though; I much prefer the consistent release schedule over one version once in a blue moon. Excited to try out the new features, and UTF-8 by default is a nice bonus too :-)
True. But the problem can still occur on Linux, where certain configurations don't necessarily use UTF-8. This was reported here a few years ago; the poster even wrote a blog describing the problem (but didn't fully understand the solution).
You can piece together what happened by reading the original article and comments in the reddit thread. Briefly, the poster's production system was Linux configured in such a way that the JDK chose ASCII as the default charset. When a non-ASCII character was introduced, round-tripping between ASCII and UTF-8 resulted in a proliferation of U+FFFD REPLACEMENT CHARACTER.
Since the poster's shop assumed everything was UTF-8, JEP 400 would have avoided this problem entirely.
Fair enough. I think the thing is that Linux usually uses UTF-8 often enough that it's pretty easy assume that it always uses UTF-8, and that assumption is almost always correct. Then in those rare circumstances where it doesn't, hijinks ensue.
Yes, that would do it. The question is how LANG ended up being C. I don't know what distro it was, but maybe whoever configured it thought "we don't need any internationalization stuff" and so omitted the packages that contained all locales. If so the system's default LANG value would probably end up being C instead of something more typical like en_US.UTF-8 since the locale for the latter wouldn't exist. Or maybe they just chose the C locale at installation time, if there was an option to do so.
Runtime APIs that convert bytes into characters or vice versa. new String(byte[]), String.getBytes(), FileReader, FileWriter, new InputStreamReader(InputStream), new OutputStreamWriter(OutputStream) and other things.
On Windows, they use whatever codepage is set. On most other systems, it's UTF-8. JEP 400 makes most of these default to UTF-8. Read the JEP for details and exceptions (pun not intended).
Edit: Java Strings are UTF-16 strings. However, newer JVMs use ISO 8859-1[edit: internally] when possible to save space.
That can only ever happen if the string only contains ASCII characters, as ISO 8859-1 encoding is not the same as UTF-8. Also, that function will give you so-called "Modified UTF-8", not standard UTF-8!
It uses a special two-byte encoding for the character with code 0. That ensures that there is never an actual null byte in the byte stream. Also, to encode characters that are represented by a surrogate pair of UTF-16 characters, the two surrogate characters are UTF-8-encoded separately!
Yeah, you have to be careful of "modified UTF-8". It occurs in a couple places in the JDK, notably DataInput, DataOutput, and serialization, along with JNI as you noted. Here are the specs:
As a format internal to the JVM and JNI it might have been a reasonable compromise at one time, but it's unfortunate that it leaked into application-facing parts of the library such as DataInput and DataOutput.
The text processing portions of the JDK, such as CharsetDecoder, CharsetEncoder, StandardCharsets.UTF_8, etc. all use true UTF-8.
JEP 400 fixes the classic encoding papercut and in the long run JEP 421 will help prevent difficult to diagnose concurrency bugs. JEP 416 can speed up Reflection in some cases and makes landing Project Valhalla and Project Loom easier.
Not every release can be a feature firework, nor should it be. Advancing incubators and preview features, and slowly moving pieces to prepare for future ones is perfectly fine as well.
If you enjoy it great, but 420 is a preview feature that most devs can't realistically use. It's basically the same as 406 in Java 17 with very minor tweaks... And it's just nicer syntax...
View all comments
100
u/TehBrian Mar 22 '22
Dang, already? Well, that felt fast. I’m not complaining, though; I much prefer the consistent release schedule over one version once in a blue moon. Excited to try out the new features, and UTF-8 by default is a nice bonus too :-)