So far we have used UTF-8000 to encode codepoints, aka non-negative integers, aka unsigned integers. I have come up with a couple of modified interpretations of the content bits which allow us to encode the entire integers, aka the signed integers.

We make use of a marvelous bijective mapping between the signed integers and unsigned integers called the zigzag function that remains a bijection when restricting to the respective n-bit ranges. We use this as a final layer at the beginning/end of the standard UTF-8000 encoding/decoding procedure.

Source

ZigZag encoding from Protobuf by Google: Protocol Buffers Documentation / Encoding

Myself: This seems like the perfect extensible solution for how to encode signed integers on top of UTF-8000.

TLDR / Examples

The code unit structure is identical to UTF-8000. The content bits correspond to the image of the zigzag function.

The ZigZag Function

The zigzag function maps from the signed integers to the unsigned integers.

If z ≥ 0 then zigzag(z) = 2 * z = (z << 1)

If z < 0 then zigzag(z) = -2 * z - 1 = -(z << 1) - 1 = ~(z << 1)

       ______________
      /  __________  \
     /  /  ______  \  \
    /  /  /  __  \  \  \
   /  /  /  /  \  \  \  \
  -4 -3 -2 -1  0  1  2  3
   \  \  \  \_____/  /  /
    .  \  \_________/  /
     .  \_____________/
      .
              

The ASCII art above illustrates the enumeration of the preimage of zigzag, showing it zigzagging between positives and negatives. This should make it clear how after 2^N steps we have covered exactly the range [-2^(N-1), 2^(N-1)).

For example the preimage of the 7-bit unsigned range [0, 128) is the 7-bit signed range [-64, 64).

Branchless ZigZag

If one is dealing with fixed-width integers, for example mapping from int32_t to uint32_t, one can create a branchless version of zigzag, wow! CPUs like branchless code.

With this example zigzag(z) = (z << 1) ^ (z >> 31), where ^ here is the C bitwise-xor operator.

If 2^31 > z ≥ 0 then (z >> 31) = 0, because we have filled the register with the highest bit of a non-negative signed number, 0. Thus zigzag(z) = (z << 1) ^ (z >> 31). Okay, nothing special?

But if -2^31 ≤ z < 0 then (z >> 31) = -1, because we have filled the register with the highest bit of a negative signed number, 1. Aha, so to achieve the bitwise complement, ~(z << 1), we can bitwise-xor with this -1. Thus zigzag(z) = (z << 1) ^ (z >> 31).

Properties

Self-synchronization, self-punctuation, and arbitrary code unit length have the same conclusion as base UTF-8000. Properties that differ are discussed below.

Encoded Range

The content bit counts work the same as they do for UTF-8000. Below is a summary of the ranges of integers that the content bits encode.

Small Integers, Small Code Units

UTF-8000 is really just a variable-width bit container with some nice properties. Provided that we obey the forbidding of overlong encoding, we can use the content bits as we please, encoding from an arbitrary alphabet to unsigned integer codewords that form the content bits.

The alphabet in question for us is the signed integers, ℤ. We heuristically think of magnitude as a measure of commonness. The closer an integer is to zero, the more common it is, and thus the smaller the unsigned integer that it should be encoded as, whence the shorter the UTF-8000 code unit it occupies. This is almost common sense.

We have observed that zigzag achieves this. Two's complements in a fixed-width register however does not do this, as for example in a 64-bit CPU register the number -1 is encoded as 111...[64]...11. This is not a problem for hardware like CPUs, but we are interested in efficient encoding for transmission and storage.

Modified strcmp(3) Ordering

Since the negative integers are interwoven (zigzagged) between the non-negative integers via zigzag, we lose strcmp ordering from UTF-8000. For example -1 < 0 but 00000001 > 00000000. However, being undeterred we can supersede this fact.

The purpose of strcmp(s1, s2) with respect to UTF-8000 is to quickly compare code units s1 and s2 as a proxy for comparing their contained codepoints, without having to actually decode the code units. For this modified version of UTF-8000 we wish to create a quick proxy for comparing the contained signed integers.

The only variants of UTF-8000 that can make exact use of strcmp are those whose content bits encode letters from a totally-ordered alphabet, for which there exists an order-preserving bijection between that alphabet and the unsigned integers. Since the unsigned integers has a minimum element, 0, and the signed integers (our alphabet) does not have a minimum element, no such bijection exists.

We know that zigzag(z) breaks nicely into two cases, non-negative signed integers and negative signed integers. We also know that order-preserving bijections do exist between 0) non-negative signed integers and the even unsigned integers, and 1) negative signed integers and the odd unsigned integers. We initially break our new strcmpsigned(s1, s2) function into four cases depending on the final bit of each code unit, which we know is a content bit and indicates whether the stored unsigned integer is even or odd. We can obtain these bits via b1 = c1 & 1 and b2 = c2 & 1 where c1 and c2 are the final bytes of the respective code units:

If b2 - b1 is non-zero then strcmpsigned can return that, as we are effectively comparing two signed integers of a different sign. Otherwise (1 - b1 - b2) * strcmp(s1, s2) effectively compares two integers of the same sign. We could therefore write this as:

strcmpsigned(s1, s2) = (b2 - b1) ? (b2 - b1) : (1 - b1 - b2) * strcmp(s1, s2)

That's pretty succinct! I have also assumed that strcmp is just the simple {-1, 0, +1} version.

Verdict

I like it! I'll add it to the reference implementation when I get chance. It will most likely be a flag -z for zigzag used like $ utf-8000 info -z -- -67 showing 11000010 10000101.

This zigzag variant has a big advantage over the rejected two's complement signed variant in that we don't need to change how we do overlong checking from the standard UTF-8000 method. This means that we can effectively separate out into layers: 1) the overlong checking of code units and the extracting of their content bits, from 2) the further decoding of the content bits eg to a signed integer.

The only external metadata needed when decoding a stream of UTF-8000 bytes is is this unsigned or signed?. This is no different to decoding a stream of raw bytes, or inspecting fixed-width integers stored in two's complement form in a CPU register: it's up to the programmer's use-case to know whether signed or unsigned is expected.

Could we apply some techniques from this document to also extend UTF-16? Yes, but at the cost of forbidding more codepoints from being encoded similar to the surrogate range U+D800 to U+DFFF.

UTF-16 is a bit messy in its existing two-byte and four-byte form, but we can clean this up in a mostly forwards-compatible manner by using ASCVI-on-UTF-16. We assume the use of big-endian UTF-16 in this section.

Source

Myself: Realizing that I can challenge the UTF-16 extensions proposed by UCS-X.

TLDR / Examples

Properties

Two-byte UTF-16 is just the raw binary form of any 16-bit codepoint, except for the surrogate range U+D800 to U+DFFF of size 2048 which any Unicode encoding (UTF-8, UTF-16, UTF-32) is forbidden to encode. The reason for this exclusion is because otherwise we could not distinguish 110110xx xxxxxxxx from 110110xx xxxxxxxx and 110111xx xxxxxxxx from 110111xx xxxxxxxx which you'll read about below.

Four-byte UTF-16 is two surrogate codepoints stuck next to each other, one high in the range U+D800 to U+DBFF and one low in the range U+DC00 to U+DFFF. The real codepoint that they encode is 0x10000 added to the 20 binary digit number contained in the content bits within. For example 11011000 00001000 11011100 00101101 contains 0x0202D, which then has 0x10000 added to it, to give 0x1202D. U+1202D is 𒀭, a Mesopotamian Dingir.

To expand UTF-16 indefinitely instead of being stuck with 0x110000 (1,114,112) codepoints, we employ ASCVI inside of UTF-16 surrogate pairs. UTF-8 was able to expand from 7-bit ASCII without any trouble because bytes with the highest bit set were undefined. In UTF-16 however every possible value that the content bits can take defines a codepoint. The best that we can do, so that we do not interfere with already assigned codepoints, and so that we do not malapportion too many unassigned codepoints for UTF-16K, is to constrain ourselves to a single unassigned plane, Plane 13. This plane is nice as the surrogates take the form 11011011 00xxxxxx 110111xx xxxxxxxx, the highest two bits of the second byte being zero. The codepoints U+D0000 to U+DFFFF are to be forbidden from being encoded, just as the 2048 surrogates of the Basic Multilingual Plane are.

We use these four-byte containers as the quantum for UTF-16K, which uses two or more of these quanta to extend from UTF-16. The content bits encode the codepoint's binary representation, without adding on 0x10000 for the sake of simplicity, similar to UTF-8000.

Bit Counts

Overlong checking is complicated for the jump from 4 bytes to 8 bytes, because of the 0x10000 that is added to the content bits. Either one of the highest 9 bits has a non-zero bit, or the 2^20 bit is active and one of the 2^m bits is active with 16 ≤ m ≤ 19.

Information Rate

For 2-byte UTF-16 this is technically 16 / 16 = 100%, notwithstanding the surrogate range.

For four bytes this is 20 / 32 = 62.5%, slightly worse than UTF-8.

For beyond four bytes this is (14k+1) / (4k*8) = 7/16 + 1/(32k) which approaches 7/16 = 43.75%, not great.

Below is a comparison of the efficiencies of UTF-8 and UTF-16.

UTF-16 is more efficient than UTF-8 only at encoding U+0800 to U+FFFF, aka the three-byte UTF-8 range that UTF-16 encodes using two bytes.

For ASCII, and beyond U+10FFFF (using UTF-8000 or UTF-16K), UTF-8000 is far more efficient (and less ugly).

Self-Synchronization

UTF-16K does not have self-synchronization at the byte level because UTF-16 does not. If one experiences a single missing byte then potentially the whole stream becomes corrupted.

At the two-byte level UTF-16 has self-synchronization which UTF-16K inherits. Non-surrogate codepoints are quantum; they are to UTF-16 as ASCII is to UTF-8. Surrogate pairs provide self-synchronization with their 110110 and 110111 high and low surrogate prefixes.

UTF-16K goes even deeper, using multiple surrogate pairs that usurp and shadow Plane 13, to encode arbitrary codepoints. Within the surrogate-pair self-synchronization level, within the high surrogates used to encode UTF-16K, 11011011 00xxxxxx, ASCVI is employed, whose leading bit provides self-synchronization, with 11011011 000 and 11011011 001.

Therefore overall UTF-16K exhibits self-synchronization at the two-byte level, like UTF-16.

Self-Punctuation

Inherited from ASCVI.

Compatibility

UTF-16K forbids Plane 13 of Unicode, because it has no way to encode those codepoints, instead repurposing the surrogate pairs erstwhile required to encode Plane 13 for the purpose of encoding codepoints beyond 0x110000. An important question to ask regarding forwards compatibility is what does an existing UTF-16 parser do if it meets a UTF-16K code unit?.

In short it's Plane-13-garbage-in Plane-13-garbage-out. Each surrogate pair used in encoding a UTF-16K codepoint beyond 0x110000 would be parsed separately, but with no syntactic issues. Semantically however this would cause issues with character (codepoint) counts that would count each surrogate pair as a separate character, rather than contributing towards a single character.

I believe that this may be a better solution than UCS-X's UTF-G-16 which breaks syntactic compatibility with UTF-16 by repurposing low surrogates as leading units for UTF-G-16 6-byte code units. One could argue that UTF-G-16 is better because those bytes would be replaced with a single replacement character � though I'm not convinced, as for example the default behavior of Python's bytes.decode function is 'strict', which raises an exception, not 'replace' which produces replacement characters.

Verdict

I don't care for UTF-16. The immature part of me says let UTF-16 decay and die as the short-sighted, legacy, Windows, wchar_t, non-self-synchronizing-at-the-byte-level, +0x10000, garbage that it is. But it will be around for a while, with several uses, like the Joliet Filesystem for my beloved Arch Linux ISOs grrr.

The main reason I wrote this section was to provide an alternative to UCS-X, so that we can use the same(ish) style as UTF-8000, and lest UCS-X or an even uglier idea come along.

The requirement to extend the list of codepoints that one is forbidden from encoding, to include Plane 13 or elsewhere, would not be an easily achieved feat.

For modern choices UTF-8 / UTF-8000 is undoubtedly the way to go.

In the same manner that ASCII is extended by UTF-8 and UTF-8000, UTF-32 could also be extended to be a multi-byte (32-bit chunk) encoding scheme.

Source

Myself: It seemed obvious how this follows from UTF-8000.

TLDR / Examples

We could extend UTF-32 either in the style of UTF-8000, treating the one-byte code units as a special case occupying the lower 31 bits...

...or in the style of ASCVI, with no special cases and the one-byte code units occupying the lower 30 bits.

Properties

Mutatis mutandis, the properties of UTF-8000 and ASCVI apply. We only make further remarks on a couple of properties.

Bit Counts

Predictable like ASCVI.

Information Rate

Whilst the ASCVI-style UTF-32K has an information rate of 30 / 32 = 93.75%, it is very inefficient for low-value codepoints, with the highest bytes most often being zeros. UTF-8000 has a much finer telescopic expansion mechanism at the byte level, compared to UTF-32 at a four-byte level.

Self-Synchronization

UTF-32 is not self-synchronizing at the byte level, and UTF-32K inherits this weakness. UTF-32 is only self-synchronizing at the four-byte level, similar to how UTF-16 is only self-synchronizing at the two-byte level. UTF-32K maintains self-synchronization at the four-byte level.

Endianness

Like UTF-16, and unlike UTF-8 and UTF-8000, UTF-32 has endianness, its quantum being a whopping four bytes.

Verdict

Not our greatest priority.

UTF-32 is hardly ever used for transmission or storage due to its inefficiency and endianness. Its main use is for when one decodes UTF-8 or UTF-16 to int32_t integers (UTF-32) for use inside a program.

What if ASCII were only six-bit instead of seven-bit? Would this make extending to multi-byte code units more pleasant?

Source

Myself: The intuitive derivation section of UTF-8000.

TLDR / Examples

Properties

Self-synchronization, self-punctuation, strcmp ordering, BOM support, and arbitrary code unit length have the same conclusion as UTF-8000. Properties that differ are discussed below.

Bit Counts

The number of content bits and mandatory content bits are even more predictable than those of UTF-8.

This is because one-byte code units are not special. They use the same 0 self-synchronization prefix as any first byte. The number 6 arises from every subsequent continuation byte adding on 7 more content bits, minus 1 for the longer start bits sequence.

Consequently the number of content bits stored in an n byte code unit is never a power of two, unlike with UTF-8000. This is because 6, containing 3 in its prime factorization, cannot divide into a power of two. This is not a terrible defect, but we do like powers of two.

Information Rate

ASCVI's information rate is 6n / 8n = 6 / 8 = 75%. This a constant independent of the length of the code unit.

For one-byte code units UTF-8000 (ASCII) is more efficient and versatile, storing double the number of codepoints and having an information rate of 87.5%.

For multi-byte code units ASCVI is more efficient, with UTF-8000's information rate tending downwards towards 62.5%.

Even if the US English alphabet had its 52 letters cut down to eg 27 Hebrew glyphs, or no letters at all, one would struggle to create a practical set of 64 glyphs for single-byte ASCVI. The tradeoff of ASCII being seven-bit, at the slight detriment of the information rate of multi-byte code units, seems worth it.

Byte Map

Unlike UTF-8000 which can never use the bytes 0xC0 and 0xC1, ASCVI uses all 256 possible bytes. Which of these is the advantageous behavior depends on whether one wants to do something extraordinary with those two bytes, or one wants to dissuade their use in chicanery.

Intuitive Derivation

See the intuitive derivation of UTF-8000 for how one might come up with this.

Naming

The II in ASCII reminds me of the Roman Numerals VII for seven, and ASCII is a seven-bit code. Therefore for this six-bit code we choose to use the Roman Numerals for six, VI, and name it ASCVI.

Verdict

7 bit ASCII, and UTF-8 that extends it, are very well established. One-byte ASCVI is inferior to the flexibility of ASCII, albeit this contributes to UTF-8 having a slightly lower information rate for multi-byte code units. I do not yearn for an alternate universe, or a fresh start of text encoding standards, where ASCII is six-bit instead of seven.

That being said, ASCVI is by no means inherently flawed, and we can make use of it in UTF-16K and UTF-32K.

One might be shocked to find our beloved two's complement representation of the signed integers present in the rejected ideas section. This is not about one's personal taste in representing signed integers, but technological extensibility, with which the zigzag signed variant far outshines this two's complement variant.

Source

Myself: It seemed like a good(ish) idea until I realized that the zigzag variant is better.

TLDR / Examples

We treat the n content bits of a code unit as a two's complement signed form, where the highest bit no longer has value 2 ^ (n-1) but rather -2 ^ (n-1).

The difference in code unit layout from UTF-8000 is that the mandatory content bits are downshifted by one bit.

Properties

Overlong Encoding Checking

Preventing overlong encoding requires checking that the content bits of an n-byte code unit encode an integer z from the range [ -2^(5n), +2^(5n) ) \ [ -2^(5(n-1)), +2^(5(n-1)) ). This may look complicated, but we can break it down into two cases:

For z ≥ 0, the highest content bit of both n-byte and n-1-byte code units is 0. There must be at least one 1 bit in the following bits, which are the mandatory content bits.

For z < 0, the highest content bit of both n-byte and n-1-byte code units is 1. This is to make z negative, using the -2^(5n) bit. There must be at least one 0 bit in the following bits, which are the mandatory content bits. Otherwise if all of these bits were ones, then z would be at least -2^(5(n-1)). For example the overlong encoding 11011111 10000000 encodes z = -64 which fits into a 1-byte code unit. Encoding integers below -64 requires subtracting from these content bits, which sets at least one of the mandatory content bits to zero.

UTF-8000 never uses the bytes 0xC0 and 0xC1, which is explained in the glossary section for overlong encoding. Slightly differently, this two's complement signed variant never uses the bytes 0xC0 (11000000) or 0xDF (11011111).

No strcmp(3) Ordering

Since negative integers set the highest content bit to 1, we lose strcmp ordering from UTF-8000. For example -1 < 0 but 01111111 > 00000000.

Verdict

A big downside of this two's complement signed variant is that its anti-overlong checking mechanism differs from that of UTF-8000 because of the 1-bit-downshifted position of the mandatory content bits. Consequently it is not possible to agnostically decode a stream of these bytes as though they were UTF-8000 bytes. For example, a 0xC1 byte is valid in this variant as 11000001, but is invalid in UTF-8000 as 11000001.

What if we tried to redeem this variant by proposing to move the negative bit, the -2^(5n) bit, to the stable tail end of the code unit, rather than it being at the ever-expanding head of the code unit? This could hopefully mean that we would not have to change the anti-overlong checking mechanism from UTF-8000. The long and short is that we would perchance intuitively reinvent the zigzag signed variant from first principles, which indeed leftwards bitshifts by one the signed integer that it encodes, using the lowest bit as the negative bit. Our redemption is found there.

What if we naively roll the start bits over into further bytes?

Source

Mashpoe on YouTube: Expanding the UTF-8 Character Set to Infinity

TLDR / Examples

Properties

Bit Counts

Effectively, every jump from 8k-1-byte code units to 8k-byte code units the encoding inserts another blank byte after the start bytes, by which 8 minus 1 equals 7 bits of content are gained in an ASCII-looking byte, instead of appending a continuation byte by which 6 minus 1 equals 5 bits of content are gained.

Information Rate

For an n-byte code unit the information rate is UTF-8000's information rate plus 2⌊n/8⌋ / (8n).

I'm not working it out fully, but I can tell that this leads to a sawtooth-y profile as a graph of information rate against n. Therefore, counterintuitively, longer code units can have better efficiency than shorter ones.

No Self-Synchronization

In the 8-byte code unit example, there is no way to distinguish the second byte 0xxxxxxx from an ASCII byte 0xxxxxxx. This generalizes to 8n-byte code units.

In the 15-byte code unit example, there is no way to distinguish the second byte, 11111110 from the first byte of a 7-byte code unit. This generalizes to 8n-1-byte code units.

In the 16-byte code unit example, there is no way to distinguish the second byte, 11111111 from the first byte of an 8-byte code unit. This generalizes such that if one seeks to any 11111111 byte, one has no idea if this is the first byte of a code unit or not.

This list is non-exhaustive.

Self-Punctuation

This is the property that Mashpoe clearly prioritized preserving, however the approach was too myopic and did not lead to preserving other properties of interest.

Patented

Mashpoe jokes (?) in the video that he owns the patent to this encoding scheme.

Regardless of whether he is joking or not, I nonetheless find it reprehensible that someone could (at least try to) copyright / patent the correct way to extend UTF-8. It would be like trying to copyright the right solution to a mathematics equation, or a prime number! Therefore I am being quite loud in the copylefting of UTF-8000 in the licensing section. Everyone benefits from shared, free-as-in-freedom, open ideas.

Verdict

The loss of self-synchronization is a fatal detriment.

The formula for the number of content bits has predictable but irritable jumps, which lead to counterintuitive information rates.

TLDR / Examples

Use up to 7 bytes to encode up to 36 bits of information in the sane way, in order to encode 32-bit integers (and a little beyond). To encode 64-bit integers, use a special-case fixed 13-byte code unit starting with 11111111.

Since the n-byte code units with n < 8 are the same as UTF-8000 we shall mostly only discuss the 13-byte code units.

Source

Perl5 on GitHub: utf8.h

A comment reads: A note on nomenclature: The term UTF-8 is used loosely and inconsistently in Perl documentation ... perl uses an extension of UTF-8 to represent code points that Unicode considers illegal..

TLDR / Examples

Properties

Bit Counts

There are no code units of length 8, 9, 10, 11, or 12. Nor are there any of length 14 or beyond.

Why 13 Bytes Instead Of 12?

Encoding the maximum possible signed 64-bit integer, 0x7FFF_FFFF_FFFF_FFFF, in Perl utf8 returns 11111111 10000000 10000111 ... 10111111. Even if the maximum possible unsigned 64-bit integer, 0xFFFF_FFFF_FFFF_FFFF, were encodable it would fit into the lower 11 bytes. So why use 13 bytes instead of 12, with the second byte always 10000000?

My guess is that the designers were going to use a UTF-Infinity style 11111111 11111000 opening, but then they recognized that they would lose self-synchronization because the second byte looks like the start of a 5-byte utf8 sequence. Therefore they swapped the second byte to a 10000000. They could have removed it and used 12 bytes, but that would preclude any future reunification with e.g. UTF-8000, which with 12 bytes has only 5 * 12 + 1 = 61 content bits which is less than 64, but with 13 bytes has 5 * 13 + 1 = 66 which is sufficient.

Information Rate

On the surface 13-byte, 72-bit code units have an information rate of 72 / (13*8) = 72 / 104 = 69%, which is better than that of UTF-8000 (62.5%).

However as only 64 of those 72 bits are used in encoding 64 bit numbers, with the whole of the first continuation byte never being used, the information rate is closer to 64 / (13*8) = 64 / 104 = 61.5%, which is worse than UTF-8000.

Self-Synchronization

This is effectively the same as UTF-8000. All continuation bytes have a 10 self-synchronization prefix, and the 13-byte start byte 11111111 has a 11 self-synchronization prefix.

Self-Punctuation

The first byte of a 13-byte code unit being 11111111 characterizes it as a special case, providing self-punctuation. This is similar to ASCII being a special case with its characterizing prefix of 0 in the highest bit.

Not Infinitely Extensible

Because Perl only supports up to 64-bit numbers without a specialized bigint module, it was sensible of them to cap their extension of UTF-8 to a finite number of bytes. It's not the prettiest however, and I'm not sure why they chose 13 bytes when 12 would suffice. CPU alignment if they don't store the predictable start byte of all 1s?

Verdict

Inextensible, providing only one special case beyond 7-byte UTF-8 to encode 64-bit numbers, and is thus not widely known or supported.

Also, they refer to it as utf8 instead of UTF-8 (the correct way), though this is pedantry.

Owl suggests a corrected version of UTF-8 with many radical changes. Many of these are opinionated, such as removing most control codes from C0. Many are technical, such as precluding the concept of overlong encodings similarly to UTF-16, by an n-byte code unit decoding to the binary number stored in the content bits added to the upper bound of codepoint values from n-1-byte code units.

Even notwithstanding the established dominance of UTF-8, I still disagree with almost everything in the document. But, he does come the closest to discovering the structure of UTF-8000's code units.

Source

Owl's Portfolio: Corrected UTF-8

TLDR / Examples

If Owl had specified the second-highest bit of his continuation start bytes to be a 0 instead of a 1 then he would have beat me to UTF-8000! So close, but so far.

Properties

No Self-Synchronization

In the 8-byte code unit example, there is no way to distinguish the second byte 110xxxxx from the first byte of a 2-byte code unit 110xxxxx. This generalizes beyond just 8-byte code units. This specific example could also encode 11000001 (0xC1), which UTF-8 cannot, which may trip UTF-8 compatibility stress-tests.

BOM Collision

Owl acknowledges that his extension may lead to issues with the UTF-16 BOM, as (presumably?) his extension permits 11111111 11111110 (0xFF 0xFE), the little-endian UTF-16 BOM.

Verdict

Owl acknowledges leaving that extension for the future with respect to going beyond the 6-byte old RFC 2044 version of UTF-8, showing humility and acknowledging his design's flaws.

I don't wish to dunk on his document too hard, but I'm greatly relieved that he failed to derive the infinite extension mechanism. It's not just for my ego's sake, but because I do not wish for UTF-8(000) to be associated with all the other junk in his specification.

Why bother publishing this now and making so much noise? As of Unicode Version 17.0, September 9th 2025, only 299,448 of 1,114,112 (27%) codepoints have been designated.

• It is a great exercise in coding theory.
• Nobody else seems to have figured it out, as only worse rejected alternatives have been previously proposed.
• If we wait until we run out of codepoints, one of those rejected alternatives may be hastily implemented just because it already exists. Granted, compatibility with UTF-16 would be broken, and first codepoints with five and six byte UTF-8 representations as per RFC 2044 could be satisfactory without needing UTF-8000.
• The current upper bound of U+10FFFF on codepoints is entirely due to UTF-16's maximum capacity. UTF-16 and wchar_t is legacy Windows tech. If the future demands a larger set of codepoints then we should allow ourselves to not be held back.
• Who doesn't like freedom, the ability to encode any integer (unsigned or signed) that we want to?
• I feel in charge of the intellectual property to an extent, and as such I have copylefted it, rather than allowing a tyrant to (re-)discover it and publish it on their restrictive terms.
• Fortune and glory. I figured this out myself in the era of the rise of AI. I'll settle for the credit lol.
• I will be submitting this work to 3b1b's Summer of Mathematics Exhibition 2026.

Verdict

Publish.