[1.2.1] Text: ASCII & Unicode

How Computers Represent Text

Every time you type a message, write a document, or browse a website, computers are working behind the scenes to represent text as binary data. Unlike numbers, which have a natural mathematical relationship with binary, text characters like letters, punctuation marks, and symbols need a special system to convert them into the 0s and 1s that computers understand. This process relies on character sets - standardised mapping systems that assign each character a unique binary code.

Understanding how text is represented in computers is fundamental to computing. Without agreed standards for representing characters, computers from different manufacturers could not communicate, files could not be shared, and the internet as we know it would not exist. The evolution from early character sets like ASCII to modern Unicode reflects the growing need for computers to handle not just English text, but all the world's languages, symbols, and even emoji.

Why Computers Need Character Sets

The Binary Challenge

Computers can only process binary data - patterns of 0s and 1s. When you press the letter 'A' on your keyboard, the computer does not inherently understand what 'A' means. Instead, it needs a systematic way to convert that character into a binary number, store or process it, and then convert it back to 'A' when displaying it on screen. This conversion requires an agreed standard that both the sending and receiving systems understand.

Imagine trying to send a text message without a common language or code system. The sender might decide 'A' equals 00000001, but if the receiver thinks 00000001 means 'Z', communication breaks down. Character sets solve this problem by providing universally agreed mappings between characters and binary codes, ensuring that everyone interprets the same binary pattern as the same character.

The Need for Standardisation

In the early days of computing, different computer manufacturers used different character encoding schemes, making it impossible to share data between systems. A file created on one computer might display as gibberish on another. The development of standardised character sets like ASCII solved this critical problem, enabling computers to exchange text reliably. Today, standards organisations maintain these character sets to ensure global compatibility and support for new symbols and languages.

ASCII: The American Standard Code for Information Interchange

What is ASCII?

ASCII (pronounced "as-key") is a character encoding standard developed in the 1960s. It was designed primarily for English text and includes the letters A-Z (both uppercase and lowercase), digits 0-9, common punctuation marks, and special control characters. ASCII originally used 7 bits to represent each character, providing 128 different possible codes (2⁷ = 128). Later, extended ASCII used 8 bits, offering 256 characters (2⁸ = 256).

The ASCII system assigns each character a unique decimal number (called its ASCII code or ASCII value), which computers store as binary. For example, the uppercase letter 'A' has ASCII code 65, which is 01000001 in binary. When you type 'A', your keyboard sends this binary code to the computer, and when displaying 'A', the computer converts 01000001 back to the letter 'A' on screen.

ASCII Character Ranges

ASCII organises its 128 characters into distinct groups, each serving a specific purpose:

Key ASCII Examples

Understanding some key ASCII codes helps illustrate how the system works:

• Space (32): 00100000 - represents the space character between words
• '0' (48): 00110000 - the character '0', not the number zero
• '9' (57): 00111001 - the character '9', not the number nine
• 'A' (65): 01000001 - uppercase A
• 'Z' (90): 01011010 - uppercase Z
• 'a' (97): 01100001 - lowercase a
• 'z' (122): 01111010 - lowercase z

Notice that uppercase and lowercase letters are separated by exactly 32 positions. This is intentional - it means you can convert between cases by flipping a single bit. The difference between 'A' (65) and 'a' (97) is 32, which in binary means toggling bit 5.

Control Characters

The first 32 ASCII codes (0-31) are control characters that do not represent printable text. Instead, they control how text is formatted or transmitted. For example, code 10 (Line Feed) moves to a new line, code 13 (Carriage Return) returns the cursor to the start of a line, and code 9 (Tab) creates horizontal spacing. These characters originated from mechanical typewriter controls but remain important in modern computing for formatting text documents and data files.

Extended ASCII

The original 7-bit ASCII only provided 128 characters, which was insufficient for many languages and applications. Extended ASCII uses 8 bits (one full byte), providing 256 characters. The additional 128 characters (codes 128-255) include accented letters (é, ñ, ü), currency symbols (£, ¥), box-drawing characters, and other specialised symbols. However, different extended ASCII standards assign different characters to these codes, causing compatibility problems when exchanging files internationally.

Limitations of ASCII

While ASCII served English-language computing well for decades, it has significant limitations:

• English-centric: ASCII was designed for English and cannot represent characters from most other languages
• No support for non-Latin scripts: Arabic, Chinese, Japanese, Hebrew, and many other writing systems cannot be represented
• Limited symbols: Modern communication needs mathematical symbols, currency signs, and emoji that ASCII cannot provide
• Extended ASCII confusion: Multiple incompatible extended ASCII standards created problems when sharing files
• Insufficient for global computing: As the internet connected the world, ASCII's limitations became increasingly problematic

Unicode: A Universal Character Set

What is Unicode?

Unicode is a modern character encoding standard designed to represent text from all writing systems in the world, plus mathematical symbols, emoji, and other characters. Developed in the 1990s by an international consortium, Unicode assigns a unique number (called a code point) to every character, regardless of platform, program, or language. Unicode currently defines over 140,000 characters covering 150+ scripts and symbol sets.

Unlike ASCII's 128 or 256 characters, Unicode can represent over 1.1 million different characters (though not all code points are currently assigned). This vast capacity ensures Unicode can accommodate all existing languages plus room for future additions, including historical scripts and newly invented symbols. Unicode has become the global standard for text representation, used by operating systems, programming languages, databases, and web browsers.

Unicode Code Points

Unicode assigns each character a unique code point, written as U+ followed by a hexadecimal number. For example:

• A: U+0041
• a: U+0061
• € (Euro sign): U+20AC
• 🙂 (Emoji): U+1F642
• 中 (Chinese character): U+4E2D
• א (Hebrew letter): U+05D0

The first 128 Unicode code points (U+0000 to U+007F) are identical to ASCII, ensuring backward compatibility. This means any valid ASCII text is also valid Unicode, making the transition from ASCII to Unicode straightforward for English-language systems.

Unicode Encoding Schemes

While Unicode defines code points for characters, it does not specify exactly how those code points should be stored as bytes in computer memory. Several encoding schemes exist to convert Unicode code points to binary data:

UTF-8 (8-bit Unicode Transformation Format)

UTF-8 is the most common Unicode encoding on the internet. It uses a variable number of bytes to represent each character - between 1 and 4 bytes depending on the character. ASCII characters (U+0000 to U+007F) require only 1 byte in UTF-8, making it highly efficient for English text. More complex characters require 2, 3, or 4 bytes.

UTF-8's variable-length design makes it space-efficient for text that is primarily ASCII (like English) whilst still supporting the full Unicode character set. This efficiency explains why UTF-8 has become the dominant encoding for web pages, email, and text files.

UTF-16 (16-bit Unicode Transformation Format)

UTF-16 uses 2 or 4 bytes per character. Most common characters (including all characters in major modern languages) require 2 bytes, whilst less common characters require 4 bytes. UTF-16 is more efficient than UTF-8 for languages like Chinese or Japanese that use many characters outside the ASCII range. Windows operating systems and Java internally use UTF-16 for text processing.

UTF-32 (32-bit Unicode Transformation Format)

UTF-32 uses exactly 4 bytes for every character, making each character the same size. This fixed-length encoding simplifies some processing operations because you can instantly calculate a character's position in a string. However, UTF-32 is wasteful of storage space - an English document would use 4 times as much space as UTF-8. UTF-32 is rarely used for storage or transmission but is sometimes used internally by programs for text processing.

Why Unicode Matters

Unicode solves the critical problems that ASCII could not address:

• Global language support: Unicode can represent text in any language, enabling true multilingual computing and global communication
• Consistent encoding: One standard replaces dozens of incompatible local character sets, eliminating encoding problems when sharing files
• Rich symbol support: Mathematical symbols, currency signs, musical notation, and emoji are all standardised
• Historical preservation: Ancient scripts like Egyptian hieroglyphics and Linear B can be digitally preserved and studied
• Future-proof: With space for 1.1 million characters, Unicode can accommodate new symbols and writing systems for decades to come

Practical Applications and Examples

Web Pages and HTML

Modern web pages use Unicode (typically UTF-8) to display multilingual content. The HTML meta tag <meta charset="UTF-8"> tells browsers to interpret the page as UTF-8 encoded. This allows a single web page to seamlessly display English, Chinese, Arabic, emoji, and any other Unicode characters without special encoding tricks. Before Unicode, websites needed complex workarounds to display multiple languages on the same page.

Programming and Databases

Modern programming languages like Python 3, Java, and JavaScript use Unicode as their native string format. This means variables can contain text in any language without special handling. Databases also store text fields in Unicode, allowing applications to support global users without redesigning data structures. When a user in Japan, Egypt, or Brazil enters their name, the database stores it correctly using Unicode.

Email and Messaging

Email standards now require Unicode support, enabling people worldwide to send messages in their native languages. International email addresses (using characters beyond ASCII) are supported through Unicode. Messaging apps like WhatsApp and Telegram rely on Unicode to display emoji, which have become a universal form of expression understood across language barriers.

Operating Systems

Modern operating systems (Windows, macOS, Linux) use Unicode for all text operations. File names, folder names, system messages, and application interfaces can all contain Unicode characters. This allows users in any country to use their computer in their native language, with file names and folders that make sense in their cultural context.

Comparing ASCII and Unicode

Real-World Impact

Internationalisation (i18n)

Unicode enables internationalisation - the process of designing software that can easily adapt to different languages and regions. Before Unicode, creating multilingual software required maintaining separate versions for different markets, each using different character sets. Today, Unicode allows a single codebase to support users worldwide, dramatically reducing development costs and improving global accessibility.

Digital Inclusion

Unicode promotes digital inclusion by ensuring people can use computers in their native languages. Communities speaking minority languages can now create digital content, develop software, and participate in online discussions using their own scripts. This preserves cultural heritage and ensures technology serves all of humanity, not just English speakers.

Search and Information Retrieval

Search engines like Google must handle queries in any language and match them to relevant content. Unicode provides the foundation for multilingual search, allowing users to search in their native language and find results from around the world. Without Unicode, building truly global information systems would be impossibly complex.

Looking Forward

Unicode continues to evolve, with new characters added regularly. Recent additions include emoji variations (different skin tones), gender-neutral emoji, and characters for newly recognised minority languages. The Unicode Consortium (the organisation maintaining the standard) reviews proposals for new characters, balancing the need for comprehensive coverage with concerns about code point allocation and compatibility.

As computing becomes increasingly global and diverse, Unicode's role as the universal character encoding standard becomes even more critical. Understanding how text is represented in computers - from the early days of ASCII to modern Unicode - provides essential insight into how digital communication works and why standards matter in technology.