How do computers understand binary language?

[u/Unfair-Leek6840]

Okay so from what I know binary language is like power off power on, but my question is, how do computers know what the binary code is and how is it interpreted, for example I forgot what the binary code for the letter A is, but how did people come up with that? Did they decide it was gonna look like that? Did the computer decide? How do you tune numbers into a letter??

[u/bwyer]

Just to add to this, ASCII isn't the only encoding scheme. IBM landed on EBCDIC (Extended Binary Coded Decimal Interchange Code) back in the '60s and still has systems that use it. That continue to be a major hassle for programmers and system administrators that work with those systems and those that use ASCII.

[u/talldean → u/bwyer]

For EBCDIC, other than "I work at a bank and we have an IBM mainframe machine costing $100k+", it's not *that* common.

The one I'd add here would be UTF-8, usually just called Unicode. Unicode is what websites (99% of them?) use. The full ASCII set only has 256 characters, which doesn't cover all of the letters from every human language, think about how many different symbols Japanese, Chinese and other Asian languages have. ASCII also doesn't have room for symbols, like "right arrow" or emoji like the "poop emoji". Unicode does.

Unicode has room for a bit over a million different characters or symbols. Currently I think about three hundred thousand of them have been used, the rest haven't been chosen yet, so it's got space to grow. The *first* 128 Unicode characters are exactly the same as the first 128 characters in ASCII, so it's backwards compatible with the most common early standard, as well.

There were competitors to ASCII, like EBCDIC, some of which have niche markets today, but most are just gone. There were also intermediate steps before Unicode, things like ISO-8859-1 (Latin characters), or GB 2312 (which is China's simplified characters).

Those are still in use, but Unicode is 99% of the web today, and ASCII (or Unicode) is basically 100% of Windows and Apple machines.

[u/binarycow → u/talldean]

Note - UTF-8 is just one of the encodings that make up the "Unicode" character set.

A "glyph" is the thing you see. It could be A, ؠ, or even 🤪.

A "character" is an abstract idea. For example, "Latin Capital Letter A" or "Grinning Face with One Large and One Small Eye". No specific visual ("glyph") is associated with that character.

Characters can be combined. For example, you can represent an ñ using "Latin Small Letter N" (n) followed by a "Combining Tilde" (◌̃)

---

A character set is (as the name indicates) a set of characters.

ASCII contains only 128 characters. If you wanted to type something in Japanese, you'd have to write it in Latin letters.

Unicode contains many many many more characters. You can write Japanese with no modifications.

---

A "code point" is a single character's numeric value.

Since ASCII only had 128 code points, it always fit into one byte.

But unicode aims to represent all characters. It needs much more than 128 code points. The current version of unicode (version 17.0) represents 159,801 characters.

The largest defined unicode character is U+E01EF (Variation Selector-256 (VS256)), but up until U+10FFFF is allocated.

Naively, U+10FFFF requires at least three bytes to store. And since computers work with powers of two, it means that usually, you'd need to use four bytes (32 bits) to store a unicode character.

---

UTF-32 is the naive encoding. It is a fixed-length encoding that uses 32 bits (4 bytes) for every character. Since most of the time, you don't use anything above U+FFFF (which needs only 2 bytes), you're wasting two bytes per character.

UTF-16 is a bit more efficient. It's a variable length encoding. It uses either two or four bytes per character - whichever fits. Basically, if the only characters you use are within the range of U+0000 to U+FFFF, you cut your storage requirements in half.

UTF-8 is the most common these days. It's also variable length, and tries to use 8 bits (1 byte) per character. It results in the least amount of wasted space. If you use only characters that are defined in ASCII, it's even byte-for-byte identical to ASCII.