From Text to Bytes

Computers only understand binary numbers (0s and 1s). To store or process text, we need a system that assigns a number to each character. As long as everyone agrees on the mapping, computers can use it to store and exchange text reliably.

This consistency is achieved through Unicode, the universal standard for text. It acts as a large lookup table that assigns a unique number, called a Code Point, to every character in almost every language. Today, Unicode defines over 150,000 characters across 161 scripts. We usually represent these code points with the notation U+XXXX (where XXXX is a hexadecimal number).

The simplest encoding would be to store every code point as a 4-byte (32-bit) integer, known as UTF-32. This handles any Unicode character easily, since even the largest code points fit in 4 bytes.

However, UTF-32 is space-inefficient for most text. For example, a simple text file of English letters (which only need 1 byte) would be four times larger than necessary. Let's consider storing "Hello" using UTF-32:

For code points that don't require 4 bytes, the computer has to pad the remaining space with zeros. So, five characters require 20 bytes, but 15 of those bytes are zeros. That's 75% waste.

For English text (which dominates the internet), UTF-32 uses four times more storage than necessary. This makes UTF-32 impractical for almost all real-world applications, so we need a more efficient encoding.

UTF-8 solves the storage problem by using fewer bytes for common characters and more bytes only when needed, instead of wasting 4 bytes on every character.

In UTF-8, different characters need different amounts of storage. ASCII characters (the basic Latin alphabet, digits, and punctuation) are so common that UTF-8 keeps them as single bytes for maximum efficiency. As characters become less common in English text (accented letters, then Asian scripts, then emoji), they get allocated more bytes. Here's how UTF-8 divides the Unicode space:

With UTF-8, "Hello" takes just 5 bytes instead of 20. English text is as compact as it was with ASCII, while still supporting every character in Unicode. Try different text in the interactive comparison below to see how the byte counts change:

Variable-width encoding creates a problem that fixed-width encodings don't have. How do you know where one character ends and the next begins? With UTF-32, every character is exactly 4 bytes, so you just read in fixed chunks of four. But with UTF-8, characters vary in length (1 to 4 bytes).

With UTF-8, the same string uses fewer bytes, but the boundaries aren't obvious:

This is the core challenge with variable-width encoding.

UTF-8 solves the boundary problem by embedding length information directly into each byte's leading bits. The number of leading 1s tells you how many bytes the character uses: a leading 0 means one byte, 110 means two, 1110 means three, and 11110 means four. Continuation bytes always start with 10, so they can never be mistaken for the start of a new character.

Here's the complete pattern:

Decoding converts bytes back into characters. For the UTF-8 byte stream from earlier, we read each byte's leading bits to determine whether it starts a new character or continues the previous one.

The boundaries are now clear: [41] [C3 A9] [E4 B8 AD] → "A", "é", "中". Each byte's leading bits told us exactly what to do, with no guessing required.

Encoding works in the opposite direction. Given a Unicode code point, we determine how many bytes it needs, then split its bits into the appropriate UTF-8 template.

Let's encode é (U+00E9, decimal 233). First, we determine how many bytes it needs by checking its value: 233 falls between 128 and 2047, so it requires the 2-byte template.

Let's trace through how "Hi 👋" gets converted to bytes.

We've now seen how UTF-8 converts any text into a sequence of bytes. Every language, every emoji, every symbol reduces to just 256 possible byte values. So why not feed these raw bytes directly into an AI model?

The issue isn't what the model can read, but how much attention it has to spend reading it. Imagine someone telling you a story by pronouncing every letter:

By the time they reach "F-O-X," you've forgotten the beginning. Your brain is so busy assembling letters that it has no capacity left for meaning.

AI models face a similar challenge. They process text sequentially, one unit at a time, and can only hold so many units in memory. This limit is called the Context Window. When each byte is a separate unit, sequences become very long. For example, "Artificial Intelligence" is just 2 words to us, but 23 separate bytes for the model. Most of its capacity gets spent on low-level details instead of meaning.

The opposite extreme is to give every word its own ID: "Apple" = 101, "Banana" = 102. This keeps sequences short, but creates new problems. First, the vocabulary becomes huge. To cover all languages, names, and technical terms, we'd need millions of IDs, consuming massive amounts of memory during training. Second, the vocabulary is rigid. When a new term like "GPT-8" is invented, the model has no ID for it and fails to process it.

So bytes give us sequences that are too long, and words give us vocabularies that are too large. The middle ground is to start with bytes and merge adjacent ones based on frequency. For instance, the bytes for 't', 'h', and 'e' appear together so often that it's efficient to merge them into a single unit, called a token. Same with common words like "apple", "run", or suffixes like "ing". Rare words like "cryptocurrency" stay as separate pieces like "crypto" + "currency".

This gives us the best of both worlds. Common text stays compact and efficient, while any new or rare word can be built from smaller, known pieces.

We now have the right concept: common words should be single units, and rare words should be split. This process of chunking text into pieces is called Tokenization, and the resulting pieces are called tokens.

In the next chapter, we'll build the algorithm that finds these optimal tokens: Byte Pair Encoding (BPE).