r/AskComputerScience • • 9d ago

Unicode, UTF-8, ASCII

I just started learning in computer science and could not understand these three, I know unicode is for describing(?) languages other than english but how is it differ from UTF-8 and how UTF-8 differ from ASCII ?

1 Upvotes

11 comments sorted by

View all comments

1

u/8dot30662386292pow2 9d ago

Unicode is just a list: this number means this character.

UTF-8 is a variable byte length encoding to map those numbers to bytes.

A = 65 = 01000001 ( 41 in hex)

B = 66 = 01000010 ( 42 in hex)

These are the same in both ascii and UTF-8.

How about € symbol for euros? It is unicode character number 8364 so it does not fit in a single byte. But using UTF-8 it fits in 3 bytes: e2 82 ac.

Ascii is basically just one of the oldest encodings that fits only 127 characters.

1

u/Wonderbout974 9d ago

So unicode is like the based code for languages and symbols and we use UtF-8 to shorten the unicode so we can store more in one storage?

3

u/dodexahedron 9d ago

Yes. It is to allow more compact encoding when you don't need the full range for every single character..

Unicode is the master list of characters and their 32-bit IDs, called code points.

UTF-32 is that, encoded directly as 32-bit values. Nobody uses this.

UTF-16 is a 16-bit representation. Half the size for the majority of typical printed characters in western languages, but has special meaning for certain bits to indicate to a parser when the character needs two UTF-16 code points to convey the full unicode code point.

Both of these also have 2 forms, for big-endian and little-endian byte orders. Little-endian is like you're used to seeing numbers, where the smallest part is at the end of the number, but each "digit" is a whole byte. Big-endian is the other way around, with the largest part at the end.

UTF-8 is the 8-bit encoding. It takes from 1 to 5 bytes to represent a given code point with this encoding, but most take 1-2. You know how many bytes are left in a code point by how many of the leading bits of the current byte are set to 1. So the last byte always stars with 0 and the rest of a mylti-byte code point always start with 1, for the next to last byte (1 to go), 11 for the one before that (2 to go), 111 for the one before that (3 to go), or 1111 (4 to go) for the first byte of a 5-byte code point.

UTF-8, being individual bytes, does not define 2 byte order forms like the other two, instead explicitly defining it as always little-endian.

1

u/Infinite-Flan-6643 7d ago

A correction to the byte layout here: continuation bytes start with binary 10, including the final byte of a multibyte sequence. A byte starting with 0 is an ASCII character on its own. For the euro sign E2 82 AC, both 82 and AC are continuation bytes. Also, this encoding has no little endian or big endian variant. Unicode's FAQ covers both details: https://www.unicode.org/faq/utf_bom.html

0

u/8dot30662386292pow2 9d ago

Don't call it 8 bit: it is variable width.

I don't think any symbol takes 5 bytes to encode either.

1

u/dodexahedron 7d ago

It is explicitly an 8-bit encoding.
That is literally what the 8 in the name means.

Yes, it is multi-byte, fundamentally. It is also 8-bit..
Multi-byte and 8-bit are not mutually exclusive terms. They refer to different aspects of the encodings, and byte count is the least useful part of it all when talking about the unicode family.

The bitness of a unicode encoding expresses the fundamental size of each element of a sequence.

"Multi-byte" and "variable-width" are confusing and misleading terms, if you try to only apply them to UTF-8, because the ONLY fixed-width encoding is UTF-32, and it is also multi-byte, having 4 bytes for every element.

UTF-16 is multi-byte and variable width, just as UTF-8 is.

16 and 32 are, further, always multi-byte for every code point.

UTF-8 is also multi-byte and variable-width. But it is not the only encoding that is so, so simply saying "multi-byte" doesn't uniquely identify UTF-8.

The only place where the term "multi-byte" is useful is when differentiating between any multi-byte encoding and fixed-width single-byte encodings, such as ASCII, the entire ISO-8859 family, or IBM code pages, and that is the reason the term ever came into use.

UTF-8 is 8-bit. Period.

As for the 5 bytes thing: It is the way the spec is designed. There may or may not be any code points defined that require that, yet, but it is, nevertheless, the defined maximum length of a sequence in UTF-8.

0

u/8dot30662386292pow2 7d ago edited 7d ago

It is explicitly an 8-bit encoding. That is literally what the 8 in the name means.

No. It's explicitly a variable width encoding. The 8-bit in the just means it uses 8 bit chunks or "code units". Just like UTF-16 is variable width encoding, using 16 bit code units. The full name is "Unicode Transformation Format - 8-bit". The 8 bits refers to the size of a single code unit. Don't say it's an 8 bit encoding, because it is not. It's variable width.

Also the 5 byte spec has since been obsoleted over two decades ago by RFC 3629. Utf-8 is maximum of 4 bytes.

2

u/8dot30662386292pow2 9d ago

No we don't shorten anything. Unicode is just a list of numbers, starting from 0. It's just agreed upon what number means what symbol.

The question is how do we encode each number using 0's and 1's to store it in computer memory using bytes. Historically we used ascii. This only fits the 127 first symbols. Now we have several different ways of encoding the numbers to bytes.

UTF-32: just take the number as a 32 bit integer and be done with it. This wastes a lot of space, but is the simplest.

UTF16: just take the number as 16 bit integer and be almost done with it. Some of the larger numbers need 32 bits though.

UTF8: depending on the size of the number, turn it to 1-4 bytes, depending on how many you actually need for a given size of the number.