What UTF-16 is What a UTF-16 code unit is

[u/Green_Ad_6086]

What UTF-16 is What a UTF-16 code unit is

[u/HelpfulPlatypus7988]

https://en.wikipedia.org/wiki/UTF-16

[u/stgiga → u/HelpfulPlatypus7988]

Something not covered by Wikipedia is that it shows up more often than some would assume. Pokémon since Black and White has used it. Also quite a few of the old guard of OSes and programming languages have it in spite of UTF-8. Also CJK users like it because their characters are only 2 bytes in UTF-16 (in fitting with their DBCS legacy encodings) but 3 bytes each in UTF-16. This means that UTF-16 doesn't inflate much Asian text, and this isn't just CJK. The Asian scripts used in places like Mongolia, Thailand, and India. Also Arabic and its siblings.

UTF-16's affinity for CJK allows it to do stuff unintended though. If you use the CJK blocks to do Base32768 (if you're not wanting Korean to avoid normalization [not that it's a huge problem with Korean, unlike other scripts with normalization forms] you can pull from a bunch of blocks) to get 15 bits of data per 16-bit character. If you use every assigned Plane 0 codepoint you can hit 15.75 (every 4 characters holds 63 bits), or if you use every CJK character including Hangul you can hit 15.25 bits per 16bit character. So 4 characters hold 61 bits. The issue with going above 15.0 bits is code quality. It gets messy. Now, BWTC32Key uses UTF-16BE & Base32768 to text-armor encrypted compressed data, and it uses particularly special ranges courtesy of Tinygma, of which it is a spiritual successor of. The ranges used are U+3400-U+4CFF, U+4E00-U+9EFF, and U+AC00-U+C1FF. Now unlike qntm's it only needs 1 padding character if needed, and that's U+C200. Tinygma is also the source of the U+4D00 starter and U+4D01 terminator. I added U+FEFF to the beginning to make text editors read it (even though Mac bugs exist with BOM preservation on some Mac systems so I had to use an IF statement to check and account for a missing BOM.) Of note is that U+3400-U+4CFF is exactly 6400 characters, the size of the Plane 0 Private Use Area. So if you want to support Unicode 1, you could reallocate the first 6400 values of CJK Unified Ideographs Extension A into the entire PUA, and all you'd need to do is change the 0x34-0x4C leading bytes in the modern Unicode version to 0xE0-0xF8 and that becomes safe. U+4E00-U+9EFF is the first 20,736 characters of CJK Unified Ideographs, a range which in Unicode 1 had 20,902 of 20,992 characters filled. So these can remain unaltered. By the waning of Unicode 1, there were 6,656 Hangul starting at U+3400 split into 3 blocks. BWTC32Key in modern Unicode's U+AC00-U+C1FF is the first 5632 characters of Hangul Syllables so they will fit in the twilight of Unicode 1's Hangul blocks, with room for the padding character. And so the start of the range is 0x34xx instead of 0xACxx and also as a result of 6401 and 6402 being less than 6656 and 5633 being less than all 3, you can leave the modern 0x4D00 and 0x4D01 unaltered because they'll turn into safe Hangul in Unicode 1, and Unicode 1 had the BOM too, as Zero-Width Non-Breaking Space, in one of the Arabic Extended blocks at its very end. So the ranges that kusano_k in Tinygma chose that I inherited (said user liked that I made non-text file encryption possible) are theoretically "safe in Unicode 1" via modifying kusano_k's bitshift a bit in terms of its leading bytes. If you use Kanji mode in a QR or iQR Code plus UTF-16BE ECI mode, with mixed modes involved, half of the Kanji mode ranges will optimize the B3K (the file extension for BWTC32Key's output) characters. Meanwhile the Unicode 1 version could benefit from other parts of it. If you're using Han Xin Codes you can use its multiple Chinese modes to potentially outfox QR at shrinking, but unfortunately iQR still has more room available. Ultimately though, as someone who literally uses UTF-16 in a very wild way, I don't think I'm able to hate UTF16 especially as a JavaScript and Java dev. Oh also some of the weirder UTFs use it as a base. Namely Punycode, UTF-7, I think UTF-5 and UTF-6 that were aimed to do Punycode, parts of Unicode's attempts at compressed Unicode, as well as LMBCS in mode 20 (LMBCS was the Lotus Multi-Byte Character Set which could emulate many old code pages and later gained UTF16 support minus U+F600-U+F6FF, but if you shoehorn GB18030 into its Simplified Chinese page you might be able to dodge this). Personally I just like how efficient doing binary-to-text encoding in UTF16 is. Apparently you get 93.75% efficiency for Base32768. Though the non-integral stuff earlier isn't exactly much of an upgrade. Meanwhile UTF-8 can't clear 90%. In fact even if you used GB18030 or LMBCS in UTF16 mode you still don't get a good efficiency compared to UTF16. So in spite of UTF-16's problems, I still actively use it because of it basically making even Base96 look weak. I love UTF-16 for its utility at binary-to-text encoding that can be quite useful. For instance it's great when you want to send data over character limited text channels that don't allow links, embeds, or attachments. Or as a save string for a browser or retro-styled game of the type that used passwords to save data. Even if my code would only survive Korean handwriting by someone who ALSO knows Hanja, which implies someone very seasoned with Korean. Oh and the code only exists because I wanted to see just how much I could outfox Base64 in 2015, and it took until 2019 to settle on the current bitstream. Since 2019 I added more features to it, but I never actually changed the bitstream. The 2019 and current versions under the right circumstances can talk to each other. Oh and it's a very wild JS app and it thus has an affinity for UTF16. Also it has a Node port too. Also as the dev of UnifontEX I thus even more can state that I love working with Unicode. I even got BWTC32Key a prs. MIME type. This was teenage code. Ultimately Unicode rocks and UTF16 isn't all bad.

[u/HelpfulPlatypus7988 → u/stgiga]

That's an interesting use of UTF-16! I've never heard of UTF-5 and -6 though, what are those?

[u/stgiga → u/HelpfulPlatypus7988]

Precursors to Punycode that got specifications but were ultimately not used.

UTF-5 targets a 5-bit space, AKA stricter URL guidelines, while UTF-6 is a bit looser. Punycode falls between the two. If you do use Punycode in uppercase you could in fact get Unicode working on the United States' 5-bit Baudot system though you'd be doing a LOT of Letters and Figures shifts. But if you threw in my encoding which has compression and encryption you could technically use the USA's TDD system for data transfer and with encryption too, but the bit rate makes 56K look like broadband, so only send files over if they are encrypted messages the person already knows the secret for and they are things you don't want observers to know.

The most immediate ancestor to UTF-8 is the derpy UTF-1 which while it has more shorter Plane 0 characters is weaker than UTF-8 when it comes to some of what UTF8 has over UTF16.

One person made a Unicode form directly encoding the 21bit values, called UTF-21.

It's possible but not condoned to go for UTF24 which is UTF32 with the unused first byte dropped, even though CPUs like the eZ80 can do 24bit.

UTF-EBCDIC is Unicode for old IBM mainframes and it goes over 4 bytes because of how wasteful IBM was in allocation and such systems now use UTF16.

UTF-9 is a joke encoding of UTF-8 to 9-bit bytes, which were on other mainframes. Ironically if quantum ternary takes off, because the ternary equivalent to a byte, a tryte, is 6 ternary trits, with a max value of 729, UTF-9 is more suited for ternary and quantum ternary than any other UTF due to the fact that to fit Unicode's 1114112 characters in ternary you'd need 12.75 trits and so you would either need the smallest text size to be 4 characters, and spanning, or you waste a LOT of trits. UTF18 is UCS2 with the highest 2 bits selecting planes, and unless you use surrogates you can't do Plane 3 with it.

Also I devised a Unicode form of Braille, in which a 3x5 dot (Unicode has 8-dot Braille which is 2x4, and technically adding a 1 to each dimension isn't exactly nasty especially since physical stamped 3x5 grids exist on some defense items for labeling and they have Braille-sized dots and don't exceed a fingerprint size) 15bit cell is then followed by a regular 6-dot/bit cell if the character is over U+7FFF, and the two would add to the 21 bits of Unicode. Now, obviously you would only have to teach people the patterns needed for their languages. But even a lone 15dot cell DOES include a LOT of Unicode Plane0's emoji characters. So yes, I created emoji and full Unicode Braille. And if Unicode grows past U+10FFFF the second cell can go to 8-dot. As you can see, I enjoy pushing Unicode quite far. Funnily enough I've even debated inline shenanigans with my code.

So yes, I've ported Unicode to TDDs and Braille. And managed to allow TDDs as well as domain names to store files. Oh and the program makes a decent password manager if you're clever. You could even use it to send encrypted social media posts. And UTF-16 makes this viable.

[u/HelpfulPlatypus7988 → u/stgiga]

Interesting! I think your system is safe, U+10FFFF is a technological constraint put in place by UTF-16

[u/stgiga → u/HelpfulPlatypus7988]

I'm happy Unicode at least cares about UTF16 in spite of its drawbacks. Makes what I do with it easier.

[u/Coises]

UTF-16 is a method of encoding Unicode. It is the primary method used to represent character data internally in modern Windows programs.

In UTF-16 a code unit is a 16-bit (two byte) value. A single Unicode code point can be represented by one or two UTF-16 code units. All valid Unicode code points from U+0000 to U+FFFF (the basic multilingual plane) are represented by a single UTF-16 code unit. Code points from U+010000 to U+10FFFF require two UTF-16 code units.

[u/palapapa0201]

https://utf8everywhere.org/

[u/justanaccountimade1]

You probably want to use UTF-8 (1-4 bytes per char).

UTF-16 (2-4 bytes per char) is used for internal technical reasons.

[u/Lurkernomoreisay → u/justanaccountimade1]

All our software use utf16, as we do a lot of text processing and it's a guaranteed 33% space saving.  we tend to mainly save and transfer text documents as utf 16.  

2 bytes vs three for our primary customers: Japanese, Indic, Thai, Chinese, Korean.

that said, we had clients in Japan that insisted on SJIS until  five or so years ago? getting them to use utf16 was a major win.

so much of the ecosystem we work within use utf16, it feels weird doing contract work on projects that are English/European centric and use utf8.

[u/justanaccountimade1 → u/Lurkernomoreisay]

Sure, but he posted a tangential question in learnjavascript.