A Brief History of Character Encodings: ASCII, Unicode, UTF-8, and the Root Cause of Mojibake

Unicode assigns every character a unique number (a code point); UTF-8 is one way of turning that number into bytes. All mojibake — "锟斤拷", "’", and friends — has a single root cause: the encoding used to write the bytes differs from the one used to read them. Try our Unicode converter to inspect characters and code points.

Three eras in brief

Era	Approach	Capability	Limitation
1963	ASCII	128 characters, 7 bits	English only
Every country for itself	GBK (China) / Shift-JIS (Japan) etc.	Local language	Mutually incompatible; same byte, different meaning
1991–present	Unicode + UTF-8	One unified numbering for all scripts	Requires universal adoption

Character set ≠ encoding

• Unicode is a character set: it says "中" = U+4E2D and "😀" = U+1F600 — just a table of numbers.
• UTF-8 is an encoding: it defines how those numbers become bytes — 1 byte for ASCII, 3 bytes for common Chinese characters, 4 bytes for emoji. Its variable-length design makes plain ASCII text fully compatible at zero cost.
• UTF-16 and UTF-32 are the other two encodings, using fixed 2/4-byte units. Today, default to UTF-8.

How mojibake happens

Write: "中" --UTF-8--> E4 B8 AD (3 bytes)
Read:  E4 B8 AD --interpreted as GBK--> garbled text

A three-byte UTF-8 Chinese character sliced into two-byte GBK units produces classic mojibake. And "锟斤拷" comes from double-transcoding the Unicode replacement character U+FFFD.

The three-step mojibake troubleshooting routine

1. Check the actual bytes first: inspect the file's real bytes in a hex viewer; don't trust the encoding your editor claims.
2. Verify declarations match reality: HTML's <meta charset>, HTTP's Content-Type, the database table charset — declare A while writing B and garbled text is guaranteed.
3. Standardize on UTF-8: use UTF-8 across the whole pipeline (files, database, HTTP, language defaults) in new projects and you eliminate 99% of encoding problems.

Last updated: 2026-09-08