From Text to Bytes

Computers only understand binary numbers (0s and 1s). To store or process text, we need a system that assigns a number to each character. As long as everyone agrees on the mapping, computers can store and exchange text reliably.

This consistency is achieved through character encoding standards. Currently, Unicode is the universal standard. It acts as a large lookup table that assigns a unique number, called a Code Point, to every character in almost every language.

Today, Unicode defines over 150,000 characters across 161 scripts. We usually represent these code points with the notation U+XXXX (where XXXX is a hexadecimal number).

The simplest encoding would be to store every code point as a 4-byte (32-bit) integer, known as UTF-32. This handles any Unicode character easily, since even the largest code points fit in 4 bytes.

However, UTF-32 is space-inefficient for most text. For example, a simple text file of English letters (which only need 1 byte) would be four times larger than necessary. Let's consider storing "Hello" using UTF-32:

For code points that don't require 4 bytes, the computer has to pad the remaining space with zeros. So, five characters require 20 bytes, but 15 of those bytes are zeros. That's 75% waste.

For English text (which dominates the internet), UTF-32 uses four times more storage than necessary. This makes UTF-32 impractical for almost all real-world applications.

To solve this problem, we use a smarter encoding.

UTF-8 solves the storage problem with a clever idea: use fewer bytes for common characters and more bytes only when needed. Instead of wasting 4 bytes on every character, UTF-8 adapts based on the code point's size.

In UTF-8, different characters need different amounts of storage. ASCII characters (the basic Latin alphabet, digits, and punctuation) are so common that UTF-8 keeps them as single bytes for maximum efficiency. As characters become less common in English text (accented letters, then Asian scripts, then emoji), they get allocated more bytes. Here's how UTF-8 divides the Unicode space:

With UTF-8, "Hello" takes just 5 bytes instead of 20. English text is as compact as it was with ASCII, while still supporting every character in Unicode.

Variable-width encoding creates a problem that fixed-width encodings don't have: how do you know where one character ends and the next begins? With UTF-32, this is trivial: every character is exactly 4 bytes, so you just read bytes in fixed chunks of four. Let's see the difference visually.

But with UTF-8, characters have variable lengths (1, 2, or 3 bytes in this example). Looking at the same string encoded in UTF-8, we see fewer bytes total, but no obvious boundaries:

This is the core challenge: when reading a stream of UTF-8 bytes, how does a computer know where one character ends and another begins? UTF-8 solves this with a clever bit pattern system.

UTF-8's brilliant solution is to embed the length information directly into each byte. The first few bits of every byte follow a strict pattern that tells you exactly what role that byte plays. Think of it like a prefix code: before the actual character data, each byte announces "I'm the start of a 2-byte character" or "I'm a continuation of the previous character."

Here's the complete pattern. Pay attention to how the leading bits change:

The rule is elegantly simple: count the leading 1s in the first byte to know how many bytes the character uses. A leading 0 means a single byte. 110 means two bytes. 1110 means three. 11110 means four. Continuation bytes always start with 10, so they can never be mistaken for the start of a character.

Now let's apply these patterns to actually decode the UTF-8 byte stream we saw earlier. We'll go through each byte, check its leading bits to determine its role, and reconstruct the original characters. This is exactly what your computer does every time it reads a text file.

And there we have it! The boundaries are now clear: [41] [C3 A9] [E4 B8 AD] → "A", "é", "中". Notice how we never had to guess. Each byte's leading bits told us exactly what to do. This self-describing property is what makes UTF-8 so robust: even if data gets corrupted or you start reading from the middle of a file, you can always find the next valid character boundary.

We've seen how to decode UTF-8, but how does the encoding process work in reverse? When your computer needs to save a character to a file, it must convert the Unicode code point into the correct sequence of bytes. Let's walk through this process step by step.

We'll encode é (U+00E9, decimal 233). First, we check which byte range it falls into: since 233 is larger than 127 but smaller than 2048, it needs 2 bytes. Now we need to split its bits and insert them into the UTF-8 template.

Let's trace through how "Hi 👋" gets converted to bytes. This is the exact process that happens before text reaches a language model.

We have just spent this entire chapter establishing that UTF-8 is the universal standard for text. It can represent every language, every emoji, and every symbol using just 256 basic units (bytes).

A very natural question arises: Why don't we just feed these raw bytes directly into the model?

It seems like the perfect solution. It's Universal: you would never need to update your model for new languages since Spanish, Chinese, and Python code are all just bytes. It's also Tiny: the model would only need a vocabulary of 256 items, which is incredibly memory efficient compared to storing hundreds of thousands of words.

Given how elegant this solution seems, could we just train a model on raw bytes? Let's think through the trade-offs:

The algorithm that finds this middle ground is called Byte Pair Encoding (BPE). It starts with individual bytes and iteratively merges the most frequent adjacent pairs. After training on a large text corpus, common patterns like "ing", "tion", and "the" become single units called tokens, while rare words get split into smaller recognizable pieces.

In the next chapter, we'll build BPE from scratch and see exactly how models like GPT-4 convert text into the units they actually process.