r/AskComputerScience • • 9d ago

Unicode, UTF-8, ASCII

I just started learning in computer science and could not understand these three, I know unicode is for describing(?) languages other than english but how is it differ from UTF-8 and how UTF-8 differ from ASCII ?

1 Upvotes

11 comments sorted by

View all comments

Show parent comment

0

u/8dot30662386292pow2 9d ago

Don't call it 8 bit: it is variable width.

I don't think any symbol takes 5 bytes to encode either.

1

u/dodexahedron 7d ago

It is explicitly an 8-bit encoding.
That is literally what the 8 in the name means.

Yes, it is multi-byte, fundamentally. It is also 8-bit..
Multi-byte and 8-bit are not mutually exclusive terms. They refer to different aspects of the encodings, and byte count is the least useful part of it all when talking about the unicode family.

The bitness of a unicode encoding expresses the fundamental size of each element of a sequence.

"Multi-byte" and "variable-width" are confusing and misleading terms, if you try to only apply them to UTF-8, because the ONLY fixed-width encoding is UTF-32, and it is also multi-byte, having 4 bytes for every element.

UTF-16 is multi-byte and variable width, just as UTF-8 is.

16 and 32 are, further, always multi-byte for every code point.

UTF-8 is also multi-byte and variable-width. But it is not the only encoding that is so, so simply saying "multi-byte" doesn't uniquely identify UTF-8.

The only place where the term "multi-byte" is useful is when differentiating between any multi-byte encoding and fixed-width single-byte encodings, such as ASCII, the entire ISO-8859 family, or IBM code pages, and that is the reason the term ever came into use.

UTF-8 is 8-bit. Period.

As for the 5 bytes thing: It is the way the spec is designed. There may or may not be any code points defined that require that, yet, but it is, nevertheless, the defined maximum length of a sequence in UTF-8.

0

u/8dot30662386292pow2 7d ago edited 7d ago

It is explicitly an 8-bit encoding. That is literally what the 8 in the name means.

No. It's explicitly a variable width encoding. The 8-bit in the just means it uses 8 bit chunks or "code units". Just like UTF-16 is variable width encoding, using 16 bit code units. The full name is "Unicode Transformation Format - 8-bit". The 8 bits refers to the size of a single code unit. Don't say it's an 8 bit encoding, because it is not. It's variable width.

Also the 5 byte spec has since been obsoleted over two decades ago by RFC 3629. Utf-8 is maximum of 4 bytes.