ASCII, Unicode and UTF-8: Planes, Normalization and Sort Order

Where the Primer Ends and This Begins

If what you need is the short version — what a character code is, why ASCII stopped at 128 slots, what the code pages broke, and how UTF-8 turns a code point into bytes — that ground is already covered on the ASCII & Unicode converter page itself, in the guide underneath the tool. This article picks up where that summary stops. It follows four threads that only start to matter once the basics are settled: how the ASCII standard was actually published, revised and renamed across twenty-three years; how the Unicode code space is carved into planes, and what lives above the first one; why two strings that look identical on screen can fail an equality test; and what all of that does to sorting, searching, and any attempt to measure how long a piece of text is.

ASA X3.4-1963, and Everything That Came After It

The version of ASCII most people picture — 128 slots, both cases of the alphabet, the familiar punctuation — is not the version that was first approved. The American Standards Association approved X3.4 on 17 June 1963, and that first edition had no lowercase letters in it at all. The columns that would eventually hold a through z were left largely unassigned, waiting on a decision nobody had taken yet.

The 1967 revision is the one that produced the table still in use. It filled in the lowercase letters, and it placed each of them exactly one bit away from its uppercase twin: flip bit 6 on A (0x41) and you land on a (0x61). That is why changing case inside the ASCII range is a single bitwise operation rather than a lookup, an efficiency trick that outlived the hardware it was designed for. The same revision added the braces and the vertical bar, and it reworked the control column — SOM was renamed SOH, and RU was dropped outright.

That revision went out as USAS X3.4-1967, and the changed prefix is not a typo. The standards body itself kept being renamed underneath the document. It started in 1918 as the American Engineering Standards Committee, became the American Standards Association in 1928, reorganized into the United States of America Standards Institute in 1966 — hence "USAS" on the 1967 cover — and finally settled on the American National Standards Institute in 1969. Revisions kept arriving under the new name: ANSI X3.4-1977, then ANSI X3.4-1986. The sponsoring committee then changed names twice more — ASC X3 became NCITS in 1996 and INCITS in 2001 — carrying that last edition with it, first as ANSI NCITS 4-1986 and then as INCITS 4-1986, which is still the one in force, reaffirmed as INCITS 4-1986 (R2022). Six decades, one 128-row table, four different organizational names on the cover.

The network-facing document is separate, and later. RFC 20, published in October 1969, is what pinned the character set down for use across ARPANET-era protocols, and it is the citation you will meet in internet specifications rather than the ANSI edition, which is not free to read. For anything that has to name an encoding in a header or a configuration file, though, the correct string is neither: IANA's preferred charset name is US-ASCII, and the pile of aliases registered beside it — ANSI_X3.4-1968 among them — exists precisely because software has spent fifty years writing down whichever name happened to be current.

Seventeen Planes, and the Fault Line at 65,536

The Unicode code space begins at U+0000 and stops at U+10FFFF: 1,114,112 positions, filed as 17 planes of 65,536 apiece. That ceiling was not picked for elegance. It is exactly what UTF-16's surrogate mechanism is able to address, and the space was capped to match — the limit exists because of an encoding, not the other way around.

Plane 0 is the Basic Multilingual Plane, and it holds nearly everything you meet in ordinary text: the Latin, Greek, Cyrillic, Arabic, Hebrew, Devanagari and Han characters in daily use, along with the punctuation and the currency signs. Above it the filing gets stranger:

• Plane 1, the Supplementary Multilingual Plane. Scripts nobody types on a phone — Linear B Syllabary beginning at U+10000, Gothic at U+10330 — share a plane with Byzantine and Western musical notation (U+1D000 and U+1D100), with the Mathematical Alphanumeric Symbols block that supplies script and fraktur letters to mathematicians (U+1D400), and, incongruously, with the emoji: Emoticons at U+1F600, Miscellaneous Symbols and Pictographs at U+1F300.
• Plane 2, the Supplementary Ideographic Plane, and plane 3, the Tertiary Ideographic Plane. Rare and historic CJK ideographs that would not fit into the Han blocks down in plane 0.
• Plane 14, the Supplementary Special-purpose Plane. A spillover area for format control characters, and almost empty by design.
• Planes 15 and 16, Supplementary Private Use Area-A and Area-B: 131,068 positions the standard undertakes never to assign, so a vendor or a closed system can define its own characters without ever colliding with a future release.

As of Unicode 17.0, published on 9 September 2025, 159,801 characters are actually encoded — roughly a seventh of the available space, after 4,803 additions in that release alone. Everything else is unassigned, and that is the design: the plane structure is a filing system deliberately built with room left in it.

What the layout produces in practice is one hard fault line, at U+FFFF, the top of plane 0. Everything at or below that boundary fits into a single 16-bit unit; everything above it does not, and a system that stores text as 16-bit units has to spend two. Emoji are above the line. So is most of what plane 1 contains. That is why a string API designed in the 1990s, back when "16 bits is enough for any character" was still a defensible position, reports a length of 2 for one grinning face — and why the converter's table counts by code point instead of by storage unit, listing the UTF-16 pair in a column of its own so both numbers are visible at once rather than one of them being inferred.

Two Sequences, One Appearance

There is a property of Unicode that has nothing to do with encodings at all, and it tends to surprise people who believed they had finished learning: the same visible text can be more than one legitimate sequence of code points. No encoding decision affects this, because it happens a layer above encoding.

The Unicode Character Database states it outright. The record for U+00E9 in UnicodeData.txt reads, in part, 00E9;LATIN SMALL LETTER E WITH ACUTE;...;0065 0301;, and that trailing field is a canonical decomposition — a declaration by the standard that this one character and the two-code-point sequence U+0065 U+0301 are the same thing. Not similar, not close enough: canonically equivalent. Any conforming process is entitled to substitute either for the other without asking.

UAX #15 defines four normalization forms to settle the question. NFD applies canonical decomposition, expanding every precomposed character into a base character plus its combining marks in a fixed order. NFC performs that decomposition and then recomposes, which means it prefers the single precomposed code point wherever the standard defines one. NFKD and NFKC run the same two operations using compatibility decomposition, a far blunter instrument: it folds stylistic variants together, so those fraktur mathematical letters collapse into plain ASCII and the ligature fi separates into f and i. NFKC is excellent for building a search index and destructive if you apply it to text you intend to store and display.

The reason any of this leaks into ordinary applications is that different systems emit different forms without being asked. W3C internationalization guidance recommends NFC for content on the web, largely on the grounds that it is what most keyboards produce. Apple's HFS+ went the other way and stored filenames decomposed — Technical Note TN1150 says so in as many words — using rules frozen at Unicode 3.2, which makes it very nearly NFD rather than exactly it. APFS then changed policy again: it keeps the name as it was handed over and normalizes only for hashing. Move a file called café.txt between systems that disagree about this and you can finish with a directory listing that displays one name and a lookup that finds nothing, because those four visible letters are five code points on one side of the transfer and four on the other.

Sorting: Chart Order Is Not Alphabetical Order

UTS #10, the Unicode Collation Algorithm, opens by demolishing the assumption most string-sorting code rests on: where a character sits in the code charts does not specify where it sorts, and no arrangement of a universal character set could ever make a binary comparison produce correct order for every language simultaneously.

Spanish demonstrates it cleanly. The alphabet recognised by the Real Academia Española runs to 27 letters, with ñ holding its own position directly after n — so año belongs between anular and apagar. Sort by code point and it lands nowhere near there: ñ is U+00F1, 241 in decimal, while z is U+007A, 122, so a raw byte-order sort files every ñ after every z, at the wrong end of the list entirely. One caution if you reach for the textbook example: ch and ll were withdrawn as separate alphabetical entries and now sort as the ordinary letter pairs they are, so they no longer illustrate anything.

The rest of the world makes it worse rather than better. German phone-book ordering treats ä as though it were ae. Swedish places ä near the very end of its alphabet, past z, as a letter in its own right. Danish and Norwegian file æ, ø and å after z, in that order. Two of those languages use the identical character and disagree about where it belongs, which is the whole reason a correct answer needs a locale attached and not just a string. The practical rule that follows is short: never sort user-visible text with a bare comparison operator. Hand it to a collator that accepts a locale — Intl.Collator in JavaScript, ICU underneath most server runtimes, an explicit COLLATE clause in SQL — and accept that the right answer legitimately changes with who is reading it.

Searching: Three Ways an Exact Match Quietly Misses

Search inherits every problem above and contributes one of its own. Three failures account for most of the reports that begin "the record is definitely in there."

• A normalization mismatch. A query typed on a keyboard that emits NFC will not compare equal to a stored value that arrived in NFD, even though the two render identically at every font size. Normalize both sides to a single form before comparing — the incoming query as well as whatever you indexed, because normalizing only one of them fixes nothing.
• Case folding past the ASCII range. Changing case stops being a per-character operation the moment you leave the Latin basics. The German ß uppercases to SS, two letters out of one, so any case-insensitive comparison built on the assumption that length survives the operation is already broken. Turkish is harder still: it distinguishes a dotless ı from a dotted i as separate letters, so uppercasing i under a Turkish locale yields İ rather than I, and a locale-sensitive uppercase applied to an identifier or a protocol keyword will silently corrupt it. That specific hazard is why case-insensitive comparisons in protocols are specified against an invariant locale rather than the user's.
• Diacritics the user never typed. Somebody searching a customer list for "Munoz" expects Muñoz to come back. Nothing gives you that for free: it has to be built, by decomposing to NFD, discarding the combining marks, and comparing what remains. Apply that fold when building the index and to the query, and never to the value you store and display.

Length: Decide Which Question You Are Asking

"How long is this text" has four separate correct answers, and most bugs in this territory come from a function that returns one of them being read as though it had returned another.

• Bytes. Depends entirely on the encoding in use, and it is what a database column width, a network frame, or a fixed-width file layout genuinely constrains.
• Code units. What String.length hands back in JavaScript, Java and C#, since those languages hold text as 16-bit units. Dependable right up until something crosses U+FFFF.
• Code points. How many entries from the Unicode catalog are present. This is the figure the converter's table reports, one row per code point.
• Grapheme clusters. How many things a reader would point at and call a character, defined by UAX #29. This is what a person means by "twenty characters."

Those four diverge far more often than they once did. A waving hand carrying a skin-tone modifier is two code points and one cluster. An e followed by a combining acute is two code points and one cluster. A flag is a pair of regional-indicator code points and one cluster. A family emoji assembled with zero-width joiners can run to seven code points, twenty-five UTF-8 bytes and eleven UTF-16 units, while remaining one object on screen that a reader will remove with a single press of backspace. Enforce a "20 character" limit using the wrong one of those four figures and you will either cut a name apart in the middle or refuse one that was never too long to begin with.

Putting It to Work

• Normalize once, on the way in, at the boundary where text enters the system, and choose NFC unless you have a concrete reason not to. Retrofitting normalization onto a table that already holds both forms is a data migration, not a configuration change.
• Never order user-visible strings with a bare comparison operator. Give a collator a locale and let the result be correct for the person reading it.
• Work out which of the four length measures a given limit is actually protecting, then enforce that one. Byte limits protect storage; cluster counts protect people.
• Treat U+FFFF as the boundary it genuinely is. Code that has never been exercised with a character above it has never been exercised with an emoji.
• When text is not merely mis-ordered but visibly replaced by different letters, stop and change diagnosis — that is the mojibake failure, a separate mechanism with a separate fix. When the trouble is a character you cannot see at all rather than one you can, read what LF, CR and NUL actually do to a file.

Every claim above is checkable one character at a time. Paste a string into the ASCII & Unicode converter and read the code point, the block, the UTF-8 bytes and the UTF-16 units off a single row — the quickest way to find out whether the é in front of you is one code point or two, and whether the emoji you are about to store occupies one unit or a pair. UTF-8 itself, standardized as RFC 3629 in November 2003 and devised eleven years earlier by Ken Thompson against design criteria set out by Rob Pike, is the one layer in this stack that mostly stays out of your way. Everything sitting above it — the planes, the normalization forms, the collation rules — is where the remaining work actually lives.