Three bugs, one cause. A user's name arrives as a row of question marks. A string of one emoji reports a length of two. A field truncated to fit a database column ends in a broken character that no font can draw. Each looks like a separate problem and each is the same misunderstanding: treating bytes, code points and visible characters as if they were the same thing.
They are three distinct layers, and text handling only becomes predictable once you know which layer each piece of your code is working at. Most languages blur the distinction in their default string type, which is convenient right up until it is not.
What follows is the three layers, then each of the three bugs traced to the layer it comes from, then the small set of rules that prevent all of them.
Bytes, code points, graphemes
A byte is eight bits — the unit files and networks actually move. A code point is one entry in the Unicode catalogue: a letter, a mark, a symbol, identified by a number. A grapheme is what a reader would call one character: possibly one code point, possibly several combined. These three counts are equal only for plain unaccented English text, which is why so much code appears to work until it meets a real name.
UTF-8, the encoding almost everything uses now, maps code points to bytes with a variable length. An ASCII character takes one byte; most accented Latin, Greek and Cyrillic take two; most CJK characters take three; emoji take four. So a string's byte length and its code-point count are different numbers, and neither is the number a user would give you if you asked how many characters they typed.
The third layer is where it gets genuinely subtle. A single visible emoji can be several code points joined together — a base symbol plus a skin-tone modifier, or several symbols joined by an invisible connector to form one glyph. A letter with an accent may be one code point or two, depending on how the text was produced, and the two forms look identical.
Bug one: the row of question marks
Mojibake — text decoded with the wrong encoding — happens when bytes written as UTF-8 are read as something else, or the reverse. The bytes are intact; the reader is applying the wrong table. The reason it survives so long in a system is that it usually only affects a minority of records, because ASCII bytes are identical in UTF-8 and in most legacy encodings. Every English-only name passes through cleanly and only the accented ones corrupt.
The corruption is often not recoverable, and this is the part worth understanding. If the wrong decoding maps some byte sequence to a replacement character, the original bytes are gone the moment that string is written back. Data corrupted this way and re-saved cannot be repaired from the stored value — only from a backup or the original source.
The cause is nearly always a boundary with an unstated encoding: a database connection, a file read, an HTTP response without a charset, a terminal. Each of those has a default, the defaults differ, and the one that does not match is the one that corrupts.
Bugs two and three: the wrong length, the broken tail
When a one-emoji string reports a length of two, the language is telling you a count from a layer you did not mean. Several major languages store strings as UTF-16 and report length in those units, so a code point above a certain range counts as two. The number is correct for what it measures; it is simply not the number of characters, and it is not the number of bytes either. Validating a user's input against a character limit with that count rejects legitimate text and accepts text that is too long for the column.
Truncation is the same error with worse consequences. Cutting a string at a fixed count of bytes can land in the middle of a multi-byte character, leaving a fragment that is not valid text — which then breaks whatever reads it, sometimes far away and much later. Cutting at a fixed count of code points is safe at the byte level but can still split a grapheme, severing an accent from its letter or a modifier from its emoji.
The rule that follows: decide which layer your limit is about. A database column limit is about bytes. A "maximum 200 characters" shown to a user is about graphemes. A protocol field may be about code points. Measure and cut at the layer the limit belongs to, and use a library that understands grapheme boundaries when the answer needs to match what a person sees.
The rules that prevent all three
Declare UTF-8 explicitly at every boundary, and never rely on a default. The database connection, the table and column collation, the file read, the HTTP Content-Type, the terminal, the source files themselves. A default that happens to be right today is a default that changes when the code runs somewhere else — and the failure will appear as a data problem, not a configuration one.
Decode once at the edge, work with text in the middle, encode once on the way out. Most encoding bugs are a value decoded twice or not at all, which is far easier to avoid when there is exactly one place in the system where bytes become text and one where the reverse happens.
And test with text that has all three layers disagreeing. Put a name with combining accents, a CJK string and a multi-code-point emoji into your fixtures and let them flow through every path that stores, truncates, measures or displays. It costs one line in a fixture file and it turns an entire category of production bug into a failing test.
Why UTF-8's variable-length encoding is what makes 'one character' ambiguous in the first place
UTF-8 represents different characters using anywhere from one to four bytes, which means the number of bytes a string occupies and the number of characters it contains are two genuinely different numbers, not the same thing measured two ways — code that assumes one byte always equals one character, a reasonable assumption for plain ASCII text, silently breaks the moment it encounters any character requiring more than one byte, which includes the overwhelming majority of the world's actual written languages beyond basic English letters and digits.
Why a single visible emoji can be counted as one, two, or even more by different string functions
Certain emoji, and some accented characters, are represented in Unicode using a base character combined with one or more additional modifying characters, or as a single character living outside the Basic Multilingual Plane requiring a surrogate pair in UTF-16 — a naive string-length function counting raw code units can report a length of two for something that visually appears, and is understood by a human reader, as a single, indivisible character, which is exactly why 'string length' is a genuinely more complicated question than it first appears once text moves beyond basic ASCII.
Why truncating a string byte-by-byte can produce a corrupted, unreadable character at the cut point
Truncating a UTF-8 encoded string at an arbitrary byte position, without checking whether that position happens to fall in the middle of a multi-byte character's own encoding, can slice directly through that character, leaving a broken, invalid byte sequence at the very end of the truncated string that most systems display as a garbled replacement character — safe truncation has to operate on complete character boundaries, not raw byte counts, which is precisely the detail a naive `substring(0, 100)`-style truncation frequently gets wrong on any text containing multi-byte characters.
Why storing text with the wrong declared encoding produces different mangled output than storing raw bytes correctly
Text saved to a database column declared with a legacy, single-byte encoding will have any multi-byte character silently corrupted or truncated the moment it is written, since that column's storage format has no way to represent the character correctly at all — this differs from a display-only mangling, where the underlying bytes are still stored correctly but rendered wrong, in a way that matters considerably for recoverability: a display bug can be fixed by displaying the same, still-intact stored bytes correctly, while data actually corrupted at the storage layer is permanently lost and cannot be recovered by any later fix to how it is displayed.
Normalization forms and why the same name can fail to match itself
Unicode has a peculiarity that trips up even encoding-aware developers: several different sequences of code points can render as the exact same visible character. An e with an accent can be stored as a single precomposed code point, or as a plain e followed by a separate combining-accent code point that gets rendered on top of it — visually identical, but byte-for-byte different strings. A name typed once through one input method and once through another can end up in either form, and a naive string comparison between the two will report them as different even though a human reading both sees the same name.
Unicode defines normalization forms — NFC, which prefers precomposed characters, and NFD, which prefers decomposed sequences — specifically to resolve this, and the practical rule is to normalize every string to the same form, typically NFC, at the point it enters your system, before it is ever compared, hashed, searched, or stored as a lookup key. Skipping this step means two spellings of the same name can silently fail to match in a search, a login lookup, or a duplicate-detection check, and the failure gives no error message at all — it just quietly returns the wrong answer.
Where this bites hardest: sorting and search
Beyond storage and length-counting, encoding assumptions break sorting and substring search in ways that are easy to miss in testing because most test data happens to be plain ASCII. A sort function that compares strings byte by byte will order accented characters in a position that looks arbitrary to a human reader, because it is sorting by the numeric value of encoded bytes rather than by any linguistically meaningful alphabetical order — correct locale-aware sorting requires a collation-aware comparison, not a raw byte comparison.
Substring search has an analogous trap: searching for a plain 'e' inside a string that contains an accented e stored in decomposed form will not find it, because the decomposed form has no plain 'e' code point standing alone at that position — it has an 'e' immediately followed by a combining accent, and a naive substring check does not know those two code points together represent something a user would consider a match for a bare 'e' search. Search features that need to feel forgiving to real names generally normalize and sometimes even strip accents specifically to route around this.