Hacker News Reader

Comment by joshuaellinger | Hacker News Reader

ASCII is by far the most successful character encoding that computers have used. It was invented in 1963, back in the era of punch cards and core memory. Modern RAM did not exist until 1975 -- a decade later.

Unicode is the replacement, not the competitor, like 64-bit IP addresses are the replacement for 32-bit IP addresses. It was developed in the early 1990s when RAM got cheap enough that you could afford two-bytes per character.

Personally, I deal with data all the time and rarely encounter unicode. Of course, I'm in the US dealing with big files out of financial and marketing databases. In fact, I've seen more EBCDIC than UNICODE.

View on HN

>64-bit IP addresses are the replacement for 32-bit IP addresses

IPv6 addresses are 128-bit.

"Modern RAM did not exist until 1975 -- a decade later."

What does that even mean? It doesn't mean DIP packaged DRAM because my dad was buying COTS Intel 1103's in 1971 or so before I was even born. And the first "I'm gonna store one bit of data in a capacitor" was done over the pond in the .uk during WWII at their code breaking plant.

"like 64-bit IP addresses are the replacement for 32-bit IP addresses."

Um...

I just looked it up on Wiki. I remember my dad showing me core memory (after it wasn't in production) in the late 70s.

It's not even really a replacement. More of an extension. Anything that talks Unicode can talk ASCII (although not vice versa).

Again the common mistake to confuse codes and encoding.

ASCII is an encoding which can represent a very small subset of Unicode.

UTF8 is an encoding which can represent all of Unicode and it is a superset of ASCII.

UTF32 can represent all of Unicode, but is not a superset of ASCII.

Therefore, something can talk Unicode via UTF32 without being able to talk ASCII.

Joel Spolsky provides the necessary (in Joel's opinion of 2003) minimum background in, The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

http://www.joelonsoftware.com/articles/Unicode.html

But the collection of Unicode codepoints is an extension of the ASCII codepoints, isn't it? (Regardless of how they are encoded).

I think its confusing the definition of "talk". Software written to handle UTF-8 can literally simply have 7bit ASCII shoved thru them. No translation necessary, no software modification necessary or anything.

On the other hand software speaking UTF-32 isn't going to tolerate ASCII being shoved thru them. The output is going to be 1/4 the length you expected and probably a mass of random asian glyphs. You CAN write a shim that turns each 7 bit ascii char into a 32 bit UTF-32 char and it'll work 1:1 perfectly for all 128 characters in ASCII. But outta the box, no you have to write a shim.

Now if you REALLY want to confuse people, after they 100% understand ASCII, UTF-8, and UTF-32, then feed some UTF-16 into something designed to eat either UTF-32 or ASCII or UTF-8. If you understand what a byte order mark is, and why its important, and how to use it, then you're along the way to understanding UTF-16.

This was a common whine about unicode in the early days that all you've done is trade the agony of multiple extended ascii code pages for the agony of multiple formats to represent unicode... Should have just spec'd UTF-8 and no other encoding and been done with it... or maybe UTF-32.

I will say that a world where only UTF-32 exists would be a world with a lot more text compression of webserver responses and stuff like that. It wouldn't be an automatic end of the world.

Yes, but that's the distinction between character sets and encodings. A character set is just that, a set of characters. In this sense Unicode is a superset of ASCII, Latin 1, and every other pre-existing character set. Encodings map characters to byte( sequence)?s and they differ a lot in that respect.

Practically both character sets and encodings prove ample opportunities to mangle input, either by replacing unavailable characters by ? or �, or by mapping to the wrong characters because the wrong encoding was assumed (or by throwing errors because the input made no sense).

I mean when ASCII assigns a number to a character, Unicode assigns the same number to the character. UTF16 and UTF32 may encode those numbers differently from how ASCII encodes them, but they are nonetheless the same numbers.

Unicode is not an extension of EDBIC or Latin1 because characters under them have different codepoints than they do under Unicode.

[Please correct me if I'm wrong. I'm not an expert but this is how it seems to me]

> [Please correct me if I'm wrong. I'm not an expert but this is how it seems to me]

You're basically right, but the subtle point is that these comments are in response to lotsofcows who said, "Anything that talks Unicode can talk ASCII (although not vice versa)."

The implication of "talk" is that the numbers are being encoded. qznc was trying to clarify that lotsofcows's statement is only true if the Unicode system is using an encoding which is backwards compatible with 7-bit ASCII, such as UTF-8.

Thanks, I see :) My comment wasn't particularly relevant.

"Unicode assigns the same number to the character."

Extremely close but not quite right for some weird corner cases. There's either a cool feature or hideous bug in the unicode design, depends how you look at it, that lets you compose multiple unicode codes together into one glyph. So you can say, "Gimme the code for A with a bar over it" or "Gimme the code for A, now gimme the code for put a bar over the last glyph". Even worse you can stack compose characters, so you can write "A with circle on top and squiggly underneath" at least five different ways in binary that should be rendered visually identical.

There is the one true normalization technique that most people use to convert what they consider poor unicode grammar into a standard form. Some people don't use it of course. And anytime you give users freeform input who knows what kind of crud they'll feed you. So you can never really assume any unicode string is normalized unless you personally normalized it yourself. Even then theoretically two normalized strings, concatenated, might or might not be normalized anymore (although this is often not much of a problem). And this strikes substring manipulation too.

A fun source of buffer overflows is normalization can shrink OR EXPAND a unicode string, in theory. So if you use one of those languages without variable length strings, look out. Or if the language tries to take care of it for you, this leads to weird memory fragmentation. Maybe a DOS/DDOS opportunity?

Which leads to philosophical argument you must decide in your code, if two bitstreams don't match, but you get matching glyph renderings, from the program's point of view are they a match or not? Depends if what your program is trying to do I guess. Most languages have a library to handle this. Writing your own unicode handler functions is a "here be dragons" moment.

Composing characters are also a fun source of swear words if you're trying to count the number of characters in a file. Something that renders literally identically will have an identical number of characters but somewhat varying number of bytes.

Other fun philosophical arguments are if you read in a un-normalized string and output a normalized string have you changed the string? Well, both no, and yes. I'm sure there's crypto steagongraphy implications.

Google for "unicode normalization form nfc" and stuff like that.

The first page I found with a good faq was:

http://www.macchiato.com/unicode/nfc-faq

Someone below noted already that grapheme clusters are, at least for dealing with humans, a much better thing to talk about than bytes or abstract characters (code points). The statement "Unicode assigns the same number to the character." was indeed correct, though. It's just that character is such an overloaded word (meaning either byte, code point or grapheme); but for Unicode at least it is synonymous with code point.

AFAIK all Latin1 characters codepoints with Unicode too.

The ASCII coded character set is a subset of Unicode, which is not true for arbitrary codesets, even if their character repertoire is a subset of the Unicode one.

But character sets are not just sets of characters, they are assignments of abstract glyphs to integers, regardless of the storage representation of the integer.

Eep, funny how I can make the same mistake twice in ~five years. You are right, of course.

Character sets therefore could be modelled as a partial function c : Character ↛ ℕ, while the encoding is a partial function e : ℕ ↛ seq Byte. Something like that.

ASCII and Latin 1 are therefore both subsets of Unicode, even though only ASCII is a subset of UTF-8 while no comparable Unicode encoding exists for Latin 1 or any other encoding.

I know what you mean.

I've worked on automated data submissions for banks in the UK, and the insistence on fixed-width, EBCDIC encoded data files for many regulatory filings (FSA, credit rating agencies) was annoying. On the other hand, it was so easy to automate in a VBA macro that I could have quite a bit of free time.

It indeed was so widespread and lasted so long that the Arab world developed a really clever standard transliteration system. https://en.wikipedia.org/wiki/Arabic_chat_alphabet

Reply on news.ycombinator.com