Skip to main content
Computer Science

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

8
  • 13
    $\begingroup$ Nitpick/clarification: Unicode is the mapping between characters and numbers (and, well, a bunch of other related stuff necessary to reproduce written text). But Unicode itself doesn't tell anything about how to store thess numbers in binary bytes. That's where UTF-7/8/16/32 come into play. They don't say anything about which number means what, but they do specify how to convert numbers into bytes. Unicode itself doesn't have a maximum limit of characters, it's abstract. UTF does (but there's still plenty of space left). $\endgroup$ Commented Jun 14, 2020 at 23:47
  • 2
    $\begingroup$ @Vilx-: The number in the answer (about a million) is familiar. IIRC, it is intentional to ensure that UTF-8 encodings take at most 4 bytes, but it is a Unicode restriction. (16 "planes" of 64K characters each). UTF-8 has a trivial extension to 5 and 6 byte encodings. $\endgroup$ Commented Jun 15, 2020 at 9:36
  • 1
    $\begingroup$ There's one possibly-security-affecting issue with UTF-8: overlong character encoding, which may be used to circumvent lists of unallowed strings. UTF-16 surrogate characters elegantly avoided this by adding an offset $\endgroup$ Commented Jun 15, 2020 at 10:35
  • 9
    $\begingroup$ @Vilx- nitpicking on your nitpick then: Unicode is a standard that defines both the database of characters and the encoding forms UTF-8, UTF-16 and UTF-32. So the Unicode standard explicitly explains how to store "these numbers" in bytes, and UTF-8/16/32 always encode Unicode characters. The standard also defines the limit to the number of characters: "Unicode codespace: A range of integers from 0 to 10FFFF" $\endgroup$ Commented Jun 15, 2020 at 11:55
  • 2
    $\begingroup$ @MSalters If you mean that it is a UTF-16 restriction pushed up to Unicode itself (and thus the other UTFs), that's true enough. $\endgroup$ Commented Jun 16, 2020 at 11:39