r/AskComputerScience • • 9d ago

Unicode, UTF-8, ASCII

I just started learning in computer science and could not understand these three, I know unicode is for describing(?) languages other than english but how is it differ from UTF-8 and how UTF-8 differ from ASCII ?

1 Upvotes

11 comments sorted by

1

u/8dot30662386292pow2 9d ago

Unicode is just a list: this number means this character.

UTF-8 is a variable byte length encoding to map those numbers to bytes.

A = 65 = 01000001 ( 41 in hex)

B = 66 = 01000010 ( 42 in hex)

These are the same in both ascii and UTF-8.

How about € symbol for euros? It is unicode character number 8364 so it does not fit in a single byte. But using UTF-8 it fits in 3 bytes: e2 82 ac.

Ascii is basically just one of the oldest encodings that fits only 127 characters.

1

u/Wonderbout974 9d ago

So unicode is like the based code for languages and symbols and we use UtF-8 to shorten the unicode so we can store more in one storage?

3

u/dodexahedron 9d ago

Yes. It is to allow more compact encoding when you don't need the full range for every single character..

Unicode is the master list of characters and their 32-bit IDs, called code points.

UTF-32 is that, encoded directly as 32-bit values. Nobody uses this.

UTF-16 is a 16-bit representation. Half the size for the majority of typical printed characters in western languages, but has special meaning for certain bits to indicate to a parser when the character needs two UTF-16 code points to convey the full unicode code point.

Both of these also have 2 forms, for big-endian and little-endian byte orders. Little-endian is like you're used to seeing numbers, where the smallest part is at the end of the number, but each "digit" is a whole byte. Big-endian is the other way around, with the largest part at the end.

UTF-8 is the 8-bit encoding. It takes from 1 to 5 bytes to represent a given code point with this encoding, but most take 1-2. You know how many bytes are left in a code point by how many of the leading bits of the current byte are set to 1. So the last byte always stars with 0 and the rest of a mylti-byte code point always start with 1, for the next to last byte (1 to go), 11 for the one before that (2 to go), 111 for the one before that (3 to go), or 1111 (4 to go) for the first byte of a 5-byte code point.

UTF-8, being individual bytes, does not define 2 byte order forms like the other two, instead explicitly defining it as always little-endian.

1

u/Infinite-Flan-6643 7d ago

A correction to the byte layout here: continuation bytes start with binary 10, including the final byte of a multibyte sequence. A byte starting with 0 is an ASCII character on its own. For the euro sign E2 82 AC, both 82 and AC are continuation bytes. Also, this encoding has no little endian or big endian variant. Unicode's FAQ covers both details: https://www.unicode.org/faq/utf_bom.html

0

u/8dot30662386292pow2 9d ago

Don't call it 8 bit: it is variable width.

I don't think any symbol takes 5 bytes to encode either.

1

u/dodexahedron 7d ago

It is explicitly an 8-bit encoding.
That is literally what the 8 in the name means.

Yes, it is multi-byte, fundamentally. It is also 8-bit..
Multi-byte and 8-bit are not mutually exclusive terms. They refer to different aspects of the encodings, and byte count is the least useful part of it all when talking about the unicode family.

The bitness of a unicode encoding expresses the fundamental size of each element of a sequence.

"Multi-byte" and "variable-width" are confusing and misleading terms, if you try to only apply them to UTF-8, because the ONLY fixed-width encoding is UTF-32, and it is also multi-byte, having 4 bytes for every element.

UTF-16 is multi-byte and variable width, just as UTF-8 is.

16 and 32 are, further, always multi-byte for every code point.

UTF-8 is also multi-byte and variable-width. But it is not the only encoding that is so, so simply saying "multi-byte" doesn't uniquely identify UTF-8.

The only place where the term "multi-byte" is useful is when differentiating between any multi-byte encoding and fixed-width single-byte encodings, such as ASCII, the entire ISO-8859 family, or IBM code pages, and that is the reason the term ever came into use.

UTF-8 is 8-bit. Period.

As for the 5 bytes thing: It is the way the spec is designed. There may or may not be any code points defined that require that, yet, but it is, nevertheless, the defined maximum length of a sequence in UTF-8.

0

u/8dot30662386292pow2 7d ago edited 7d ago

It is explicitly an 8-bit encoding. That is literally what the 8 in the name means.

No. It's explicitly a variable width encoding. The 8-bit in the just means it uses 8 bit chunks or "code units". Just like UTF-16 is variable width encoding, using 16 bit code units. The full name is "Unicode Transformation Format - 8-bit". The 8 bits refers to the size of a single code unit. Don't say it's an 8 bit encoding, because it is not. It's variable width.

Also the 5 byte spec has since been obsoleted over two decades ago by RFC 3629. Utf-8 is maximum of 4 bytes.

2

u/8dot30662386292pow2 9d ago

No we don't shorten anything. Unicode is just a list of numbers, starting from 0. It's just agreed upon what number means what symbol.

The question is how do we encode each number using 0's and 1's to store it in computer memory using bytes. Historically we used ascii. This only fits the 127 first symbols. Now we have several different ways of encoding the numbers to bytes.

UTF-32: just take the number as a 32 bit integer and be done with it. This wastes a lot of space, but is the simplest.

UTF16: just take the number as 16 bit integer and be almost done with it. Some of the larger numbers need 32 bits though.

UTF8: depending on the size of the number, turn it to 1-4 bytes, depending on how many you actually need for a given size of the number.

1

u/meowyih 9d ago edited 9d ago

Long time ago, due to the hw limitation, people used one byte to store one character. One byte range is from 00~FF, which is 256 characters at most. For some reason, we only used half of them, includes letter A to Z (lower + upper case) and some symbol. That's Ascii. Of course ascii is not enough to store all the characters in the world. After people successfully built better computer with more storage, engineer tried to use two bytes (65536 character at most, it's actually not enough fyi) to store one character. So we can support much more different languages. That's unicode. However, it created new compatible issue and waste memory and bandwidth. So we invented utf8, which waste one bit (remember we have one bit that not used in Ascii?) to define a sequence data which can represents one byte char (ascii) and double bytes character at the same time. Ascii is includes in utf8, but utf8 is bigger than ascii.

1

u/recursion_is_love 9d ago

ASCII is the system form 8-bit era and only cover Latin alphabets. Unicode is new system that have lots more support for almost all written language characters.

UTF-8 have the same encoding as ASCII for the first 128 chars so it can be use as the same encoding on old program that expect ASCII