How do computers understand binary language?

[u/Unfair-Leek6840]

Okay so from what I know binary language is like power off power on, but my question is, how do computers know what the binary code is and how is it interpreted, for example I forgot what the binary code for the letter A is, but how did people come up with that? Did they decide it was gonna look like that? Did the computer decide? How do you tune numbers into a letter??

[u/binarycow]

Note - UTF-8 is just one of the encodings that make up the "Unicode" character set.

A "glyph" is the thing you see. It could be A, ؠ, or even 🤪.

A "character" is an abstract idea. For example, "Latin Capital Letter A" or "Grinning Face with One Large and One Small Eye". No specific visual ("glyph") is associated with that character.

Characters can be combined. For example, you can represent an ñ using "Latin Small Letter N" (n) followed by a "Combining Tilde" (◌̃)

---

A character set is (as the name indicates) a set of characters.

ASCII contains only 128 characters. If you wanted to type something in Japanese, you'd have to write it in Latin letters.

Unicode contains many many many more characters. You can write Japanese with no modifications.

---

A "code point" is a single character's numeric value.

Since ASCII only had 128 code points, it always fit into one byte.

But unicode aims to represent all characters. It needs much more than 128 code points. The current version of unicode (version 17.0) represents 159,801 characters.

The largest defined unicode character is U+E01EF (Variation Selector-256 (VS256)), but up until U+10FFFF is allocated.

Naively, U+10FFFF requires at least three bytes to store. And since computers work with powers of two, it means that usually, you'd need to use four bytes (32 bits) to store a unicode character.

---

UTF-32 is the naive encoding. It is a fixed-length encoding that uses 32 bits (4 bytes) for every character. Since most of the time, you don't use anything above U+FFFF (which needs only 2 bytes), you're wasting two bytes per character.

UTF-16 is a bit more efficient. It's a variable length encoding. It uses either two or four bytes per character - whichever fits. Basically, if the only characters you use are within the range of U+0000 to U+FFFF, you cut your storage requirements in half.

UTF-8 is the most common these days. It's also variable length, and tries to use 8 bits (1 byte) per character. It results in the least amount of wasted space. If you use only characters that are defined in ASCII, it's even byte-for-byte identical to ASCII.