3  Text Encodings and Unicode

“The hardest part of working with text is remembering that computers do not see letters—they see numbers.”

Most computer users never think about text encodings.

They write documents, send emails, browse websites, and exchange files without giving any thought to how their words are represented inside a computer.

Then one day, a document opens with strange symbols instead of readable text.

Perhaps “café” becomes

café

or quotation marks appear as

“Hello�

Suddenly, a perfectly ordinary document has become unreadable.

What happened?

The answer lies in text encoding.

3.1 Computers Do Not Understand Letters

People think in letters and words.

Computers think in numbers.

When you type the letter

A

the computer does not store the shape of the letter. Instead, it stores a numerical value that represents it.

Every character—letters, digits, punctuation marks, spaces, and symbols—is assigned a number.

A text encoding is simply a system that defines the relationship between characters and those numbers.

Without an agreed encoding, one computer cannot know how another intended to represent its text.

3.2 The Early Days: ASCII

One of the earliest and most influential character encodings was ASCII, short for the American Standard Code for Information Interchange.

Developed during the 1960s, ASCII assigned numbers to 128 characters.

These included:

• the English alphabet
• digits
• punctuation
• control characters such as carriage return and line feed

For example,

Character	ASCII Number
A	65
B	66
a	97
0	48
Space	32

ASCII was simple and efficient.

It quickly became the common language of early computers and remains the foundation of many modern systems.

Unfortunately, ASCII had an important limitation.

It was designed primarily for English.

Languages containing accented letters, non-Latin alphabets, or entirely different writing systems could not be represented adequately.

3.3 The Encoding Problem

As computing spread around the world, different countries developed their own character encodings.

Various standards attempted to support European languages, while others served Arabic, Hebrew, Cyrillic, Greek, Chinese, Japanese, Korean, and many more.

Unfortunately, these encodings were often incompatible.

The same numerical value might represent completely different characters in different encodings.

A document created on one computer could become unreadable on another if the wrong encoding was assumed.

This phenomenon became known as mojibake—garbled text produced when characters are interpreted using the wrong encoding.

Many computer users have encountered mojibake without ever knowing its name.

3.4 Unicode: One Standard for the World

The solution arrived in the form of Unicode.

Rather than creating yet another regional encoding, Unicode set out to provide a universal system capable of representing every writing system used by humanity.

Its goal was simple:

Every character should have one unique code point, regardless of language or platform.

Today Unicode includes:

• Latin alphabets
• Greek
• Cyrillic
• Arabic
• Hebrew
• Chinese
• Japanese
• Korean
• Devanagari
• mathematical symbols
• musical notation
• emojis
• and thousands of other characters

Whether you write English, Shona, French, Chinese, or Arabic, Unicode provides a common framework for representing your text.

3.5 Code Points and Encodings

Unicode introduces an important distinction.

A code point identifies a character.

For example, the letter

A

has the Unicode code point

U+0041

The smiling face 😊 has its own code point.

So does every other character.

However, code points are not the same as the bytes stored in a file.

They must still be encoded into bytes.

This is where formats such as UTF-8, UTF-16, and UTF-32 come into play.

3.6 UTF-8: The Universal Choice

Among Unicode encodings, UTF-8 has become the dominant standard.

UTF-8 has several important advantages.

It is:

• compatible with ASCII
• efficient for English text
• capable of representing every Unicode character
• widely supported on every major operating system
• the standard encoding for the modern Web

Today, most websites, programming languages, markup languages, and publishing systems expect UTF-8 by default.

If you begin a new plain text project today, UTF-8 is almost always the correct choice.

3.7 Why Encodings Matter

Many people assume that text is simply text.

In reality, software must know which encoding was used when the file was created.

Suppose someone writes a document using UTF-8.

If another program mistakenly assumes the file uses a different encoding, the numerical values will be interpreted incorrectly.

The result is garbled text rather than meaningful words.

The document itself is not necessarily damaged.

The software is simply reading the numbers incorrectly.

3.8 Encodings and Plain Text

One might wonder why a discussion of encodings belongs in a book about plain text.

The answer is straightforward.

Plain text consists of characters.

Encodings determine how those characters are represented inside computers.

Without an encoding, plain text cannot exist as digital information.

Fortunately, modern computing has become far simpler than it once was.

Thanks to Unicode and UTF-8, users rarely need to think about encodings unless they are working with older documents or legacy software.

3.9 Accessibility and Global Communication

Unicode has done more than solve technical problems.

It has made computing genuinely global.

People can now exchange documents in hundreds of languages without inventing separate encoding standards for each one.

For accessibility, this has been especially important.

Screen readers, braille displays, search engines, translation systems, and digital publishing platforms all benefit from having a common way to represent text.

Whether a document contains English, Shona, French, Japanese, or mathematical symbols, modern software can process it consistently.

Unicode allows the world’s languages to share the same digital space.

3.10 Lessons for the Textsmith

Fortunately, you do not need to memorize code points or study binary numbers to work effectively with plain text.

Instead, remember a few practical principles:

• Save new plain text documents as UTF-8 whenever possible.
• Avoid changing encodings unless you have a specific reason.
• If text appears garbled, suspect an encoding mismatch before assuming the file is corrupt.
• Modern text editors usually detect UTF-8 automatically.
• Plain text becomes truly universal only when its encoding is understood.

These simple habits will prevent many frustrating problems.

3.11 Key Ideas

• Computers store numbers, not letters.
• A text encoding maps characters to numerical values.
• ASCII was the first widely adopted character encoding but supported only a limited character set.
• Unicode provides a universal character repertoire for the world’s writing systems.
• UTF-8 is the most widely used Unicode encoding today.
• Garbled text is often caused by an encoding mismatch rather than file corruption.
• Understanding encodings helps ensure that plain text remains portable, readable, and accessible.

In the next chapter, we turn from the representation of text to a philosophy that has shaped software for decades: the Unix belief that simple text streams and small, focused programs can solve surprisingly complex problems.