Talk:UTF-8

Table should not only use color to encode information (but formatting like bold and underline)

As in a previous comment https://en.wikipedia.org/wiki/Talk:UTF-8/Archive_1#Colour_in_example_table? this has been done before, and is *better* so that everyone can clearly see the different part of the code. Relying on color alone is not good, due to color vision deficiencies and varying color rendition on devices. — Preceding unsigned comment added by 88.219.179.109 (talk • contribs) 02:26, 17 April 2020‎ (UTC)

Unicode no. of characters wrong

Unicode has 1,111,412 characters. Please make this change. FrierMAnaro (talk) 14:17, 31 October 2024 (UTC)

0x110000 is 1,114,112, but the number shown is after subtracting the 2048 surrogate halves (I disagree but the consensus was that they should not count) Spitzak (talk) 17:58, 31 October 2024 (UTC)

Indeed, the Unicode Standard explicitly states it contains 1,114,112 code points right in its introduction, but there are much fewer characters. We're just quite loose in distinguishing between code points, characters, Unicode scalar values, and not well-defined ad-hoc phrases like valid Unicode code points as currently used in the second paragraph of the article. UTF-8 does not encode "code points" or "characters" but "Unicode scalar values" (D76). There are 1,112,064 of these. Not all are assigned to characters yet; some are explicitly designated noncharacters. UTF encodings can encode them all, but there are no well-formed sequences of code units that would represent surrogate code points. The wording is grossly imprecise, but the numbers are correct. – MwGamera (talk) 23:10, 31 October 2024 (UTC)

I changed it to say "Unicode scalar values" and added a citation of the Unicode 16.0.0 standard to the reference for the number. Guy Harris (talk) 21:53, 1 November 2024 (UTC)

Surrogate halves are "code points", but they are not themselves individually "characters" in the most common meaning of the term. They're elements which can be used in pairs to encode characters. AnonMoos (talk) 18:17, 1 November 2024 (UTC)

Tooltips for code points

Can you add a tooltip? Add a tooltip to every cell of the table which shows the range of code points the byte can encode. Also add tooltips for characters beyond the 10FFFF. FrierMAnaro (talk) 07:14, 17 November 2024 (UTC)

Alternative conversion table

I have always found the conversion table a little confusing, so I made a more simple alternative.

https://x.com/LatinSuD/status/1869138590271488375/photo/1

If you like it, I (or somebody) could try to complete it and convert to SVG maybe? LatinSuD (talk) 22:09, 17 December 2024 (UTC)

We do not recommend additional media rendered as images what should really be text. Remsense ‥ 论 22:28, 17 December 2024 (UTC)

Looks kind of nice, but there is a desire to keep the table resembling the references, which just use text. Spitzak (talk) 00:30, 18 December 2024 (UTC)

Clarification on overlong encodings being a "vulnerability".

I propose something among the following lines. Overlong encodings on their own are not a vulnerability. What is a vulnerability is (in the example given with ../) trying to match raw ASCII and UTF-8 bytes and thinking they're always equivalent for the same characters, and then sending that incorrectly "sanitized" input to a file system library or something that is UTF-8 aware and will decode it. When doing pattern matching across code points, overlong encoding ceases to be a problem because all strings are already decoded.

So is not an inherent "security flaw" of UTF-8 as a format, rather it's a library bug (or an inherent flaw in merely trying to detect the string .. in paths depending on your view). This does not appear to be reflected in the current section. ThanForIllegitimateAsThisDean (talk) 15:49, 24 February 2026 (UTC)

Imagine some code that has the job of "don't allow `../` to be in the filename". It can (correctly) just look for this 3-byte pattern. If the filename is then passed to a system that has to decode the UTF-8 (perhaps because the filesystem uses UTF-16), and this one is incorrectly written, it may decode another series of 4-6 bytes into `../` and allow a blackhat to bypass the check. The standard says that it is this translator that is broken, not the checking code. Spitzak (talk) 19:39, 24 February 2026 (UTC)

You are wrong about the correct behavior. It absolutely is NOT the front ends responsibility to recognize every possible sequence that some broken backend will turn into '.' or '/'. Also you claim that it just has to "decode" the UTF-8 to do this, but that is basically saying the "decoding" has to recognize overlong encodings, along with every other possible mistake that a backend might do. This is impossible. Spitzak (talk) 19:45, 24 February 2026 (UTC)

You also seem to be confused into thinking that UTF-8 cannot be used unless it is "decoded". That is false. Its entire design is so there is a unique byte pattern for each string, and therefore testing strings for equality, concatenation, and using them to identify objects, can all be done without "decoding". Spitzak (talk) 19:47, 24 February 2026 (UTC)

For reference, overlong encodings being strictly illegal wasn't specified until well into 2003. The point was that if every string is converted to runes before matching, two options are available: Either the overlong encodings are accepted, and the match is done right anyway (since exactly one kind of rune exists for each character), or the overlong encodings are rejected and the string is declared invalid. Neither of those options are possible with just raw byte matches. ThanForIllegitimateAsThisDean (talk) 22:31, 24 February 2026 (UTC)

(That's "rune" as in "code point in Unicode".) Not all code converts UTF-8 or UTF-16 to UTF-32 when, for example, checking for "../" in pathnames or URLs, so presumably we're not assuming conversion to runes here. Guy Harris (talk) 21:32, 25 February 2026 (UTC)

Not in the specific cases where the bug occurs (at least I checked for IIS). And it wasn't really the point of correct behaviour or not I wanted to debate, just an elaboration. Perhaps my other reply was too much of a tangent on the mechanisms of processing encodings or matching, and how to handle incorrect cases. Just wouldn't prefer to label it as an "unicode vulnerability" as the IIS source and the current paragraph in the article appears to present it as. ThanForIllegitimateAsThisDean (talk) 13:55, 26 February 2026 (UTC)

I see I'm not the first to take issue with this paragraph. I was going to suggest rewriting it as:

[...] is termed an overlong encoding. Because some software libraries will decode these and others will not, they have been used to bypass simple security checks and perform attacks such as XSS^[CN] and path traversal.^{[IIS ref][Tomcat ref]} It's advised that decoders treat overlong encodings as invalid.

There's now an explicit warning about these in the current version of RFC3629, though really the authors of the mentioned software should have been validating the final value to be used, which clearly they were not. YoshiRulz (talk) 21:05, 24 February 2026 (UTC)

No, you seem to be misunderstanding and insisting that you must "decode" the UTF-8 to correctly handle it.

• The code that checks for the three bytes `../` is CORRECT
• The code that converts some other sequence of bytes into `../` is WRONG

Any description must make it clear which code is wrong. It is not "differences in decoding", it is a broken decoder. Spitzak (talk) 19:23, 26 February 2026 (UTC)

You've assumed that the vulnerable software had separate components for validating the input and adapting it for use with the filesystem, which I think is reasonable, but neglects that someone had to combine those components. And this combining was done with maybe a type check ("both functions take a u8*, that means string, perfect") but without carefully comparing the invariants of each.

Code which searched only for [ 0x2E, 0x2E, 0x2F ] would be correct under the assumption that the input was in the form of valid UTF-8 without overlong encodings. Similarly, a UTF-8 to UTF-16 converter which accepted overlong encodings could be correct. It's only when used with input of an unexpected kind that problems arise. I'd hope these invariants were documented, but even the best documentation is no guarantee that a programmer won't accidentally violate them, which is why we have things like refinement types. YoshiRulz (talk) 21:52, 1 March 2026 (UTC)

NO. By definition, a converter that accepts overlong encodings is WRONG. This reduces the number of "variants" anybody needs to consider to exactly one. Anything else is not UTF-8. Spitzak (talk) 19:37, 2 March 2026 (UTC)

As ThanForIllegitimateAsThisDean noted, the definition wasn't the same in the past (before Unicode 3.1) and so the situation described would indeed be possible historically. – MwGamera (talk) 00:38, 3 March 2026 (UTC)

Perhaps the § Overlong encodings could be phrased more categorically to state that there are simply no such overlong encodings in UTF-8, but that there were in the past and their existence made it trickier to implement some things correctly, leading to security (and other!) problems as per refs. Cf. UTR #36 § 3.1 UTF-8 Exploit. But it's not "advised" to treat them as invalid, it's a conformance requirement for Unicode (clause C10 with ref to D92). – MwGamera (talk) 01:09, 3 March 2026 (UTC)

Alright, I've let this discussion sit for a while (far too long honestly, is everybody still here?) but nothing seems to be happening anymore. At this point I'm starting to think more about what to actually write in the article. The suggestion @YoshiRulz wrote above does work as a base. Of course we'd have to replace "advised" with "mandated by the specification", though I personally put more emphasis on the treatment of overlong encodings in software libraries being inconsistent. Perhaps an example (though not too wordy) can be given in the paragraph as to how exactly this causes XSS, but just having the references there is a start. I'll leave it to you for now. ThanForIllegitimateAsThisDean (talk) 12:28, 25 March 2026 (UTC)

NO! The code to make sure "../" is not in a string needs to look for exactly ONE 3-byte pattern. This simple validator is CORRECT and is run before (or likely without ever) "converting to final result". If whatever code does "converting to final result" turns some other pattern of bytes into "../" then that code is WRONG, the validator is still CORRECT. Do not put in any text that implies otherwise! Spitzak (talk) 17:53, 25 March 2026 (UTC)

More specifically, I strongly object to "the authors of the mentioned software should have been validating the final value to be used". That is absolutely wrong. In this case "the authors must fix the 'convert to final result' step". Spitzak (talk) 17:56, 25 March 2026 (UTC)

At this moment neither of those were in any draft. The first thing that came up was the bugs being present and how they exist. The whole details of who does the reporting on overlong encodings doesn't really matter as long as something does, and the paragraph was to mention that this is not always a given in practice. ThanForIllegitimateAsThisDean (talk) 20:46, 25 March 2026 (UTC)

What is the problem with the current wording at UTF-8#Overlong encodings? It says that overlong encodings are a security problem and should be considered an error and never decoded. The OP seems to suggest that checking for the three bytes ../ is inadequate because subsequent processing by a broken library might accept an overlong encoding. Instead, a checker should decode the UTF-8 into code points, and pass the checked code points for subsequent processing. Regardless of whether that would be desirable, this article has to describe what UTF-8 is. The section is correct as far as I can see. Johnuniq (talk) 02:54, 26 March 2026 (UTC)

Have to admit my original post was poorly worded, but my problems involve how bugs occur. At this point it's only stated that overlong encodings cause bugs, which while having sources does not directly state any details. Which to me was vague because if you decode all your input universally (whether passing overlong encodings or not) you do not have those bugs at all. Pedantic or not, I still prefer a slight clarification. Same with me referring to the vulnerability being in the libraries as opposed to in overlong encodings themselves, as stated right now ("...is termed an overlong encoding. These are a security problem..."), though that is less important.

The semantics of existing libraries are not really relevant to this paragraph beyond the fact they are inconsistent as stated in the messages above (having both 8-bit clean pattern matching and actually UTF-8 aware reading that happens to pass overlong encodings, which is the crux of the vulnerabilities).

And as was also noted, it's not that passing overlong encodings merely "should be considered" an error, but it's mandated by spec that it's an error, this is also not mentioned in the current paragraph.

This doesn't exactly summarise all we were discussing, but it is the gist of my original point plus a bit more. ThanForIllegitimateAsThisDean (talk) 20:00, 26 March 2026 (UTC)

I agree with your basic sentiment which, I believe, is that security is critical and to suggest that something bad "should" be considered an error is totally inadequate from a security point of view. However, in the context of this article that wording is reasonable. Perhaps something better can be devised but generally articles do not give advice (a Wikipedia defect in a case like this). Looking at it another way, anyone relying on this article when coding a UTF-8 decoder is going to make a mess of it regardless of the wording here. Johnuniq (talk) 00:15, 27 March 2026 (UTC)

This isn't really wikipedia giving advice. "is an error" is just referring to the specification saying so, explaining how the bugs occur is just explaining it, not in and of itself giving advice. Merely providing a brief summary on what the sourced studies found (and again, how things happened) should be plenty enough for the purposes of this article. ThanForIllegitimateAsThisDean (talk) 20:01, 27 March 2026 (UTC)

I share the sentiment that the section is at best vague, not being helpful for the layman nor for a programmer interested in this class of vulnerabilities. As I see it, the current problems are that 1a) it implies overlong encodings are a security problem, rather than lead to security problems; 1b) it implies overlong encodings are something allowed by the spec, which I suppose was true of the original version, but as mentioned above the spec now explicitly prohibits their use; 2) it states that overlong encodings allow e.g. malicious JS and ../ to bypass security validations, which IMO simply needs to be copyedited needed a copyedit, now done; and most importantly 3) it states that overlong encodings lead to security problems and should therefore be considered an error and never decoded, while in fact they were always erroneous, shouldn't be decoded for spec-conformance reasons, and the security problems arose from the meeting of conformant and non-conformant implementations. YoshiRulz (talk) 00:31, 9 April 2026 (UTC)

Correct: Overlong encodings are not a security problem. But decoding overlong encodings is a security problem. The best way to avoid this security problem is to NOT DECODE. For instance if the file system only looked for the byte that is an ASCII '/' and all other bytes are used unchanged as the "file names", it would work fine. This is despite the fact that it never "decodes" those filenames, and that it does not prevent overlong encodings to be put into those filenames. The problem I have with some of this wording is that it implies you must "decode" in order to avoid errors. In fact the UTF-8 spec is written the way it is so that you don't have to decode. Spitzak (talk) 00:42, 9 April 2026 (UTC)

This hypothetical implementation of a path traversal checker isn't in the article, so the specifics of that aren't at issue. (I would argue that "not decoding" UTF-8 and interpreting it as an ASCII-like single-byte encoding is still a form of decoding, and non-conformant per later versions of the spec.) You seem to believe that we're trying to put a claim like All programs must decode UTF-8 to UTF-32 before any further processing into the article, but that's not the case. The article just needs to reflect the reality that overlong encodings are an inherent part of the encoding scheme, the spec is clear about what to do with them, and when programmers don't follow that exactly then security problems can arise. YoshiRulz (talk) 02:03, 9 April 2026 (UTC)

My proposal is fully conformant with the spec. It does not decode invalid UTF-8 sequences, since such sequences cannot contain the '/' character which is the only character it "decodes". Spitzak (talk) 03:56, 9 April 2026 (UTC)