Hacker News Reader

Comment by dsp1234 | Hacker News Reader

This code will work on all UTF-8 encoded strings… which is the bulk of the strings found on the Internet if you consider that UTF-8 is a superset of ASCII.

Note that the first set of code (and possibly the rest), only work as the space, newline, and carriage return are the 7 bit ASCII set is included in UTF-8. However, the extended 8-bit ASCII set is not, but is often included when people speak of ASCII. So for example, if the request was to remove all "Copyright Sign" symbols, which is U00A9, it would not work correctly. The UTF-8 encoding for this symbol is 0xC2 0xA9, but the code only works on individual bytes, so it would remove the A9 byte, leaving a C2 byte and then whatever byte came next. Additionally, it would hit other UTF-8 characters like the "Greek Capital Letter Omega" (Ω which is encoded in UTF-8 as 0xCE 0xA9)

tl;dr Only works for the 7-bit ASCII set, but not the common extended 8-bit ASCII sets

View on HN

It's sort of a semantic argument, but I've never once heard anyone refer to "ASCII" to refer to anything other than the 7-bit standard. There is no "extended 8-bit ASCII set". That's a blanket term (though in casual use, "code page" is more typical) for any of the literally dozens of 8-bit character encodings that overlap with ASCII in the bottom half of the encoding space.

Basically the blog post was correct and precise: it removes ASCII characters from UTF-8. There is no elaboration needed.

Again, though, the post was about using SIMD primitives to optimize what looks like a scalar problem.

A number of the 8-bit sets that embedded the ASCII 7-bit set were informally referred to as ASCII especially during the 1980s; this may have been encouraged by microcomputer BASIC interpreters using the ASC() function (named for ASCII) to return the character code of a character in the system code set, which was often an 8-bit set that embedded ASCII.

You would be surprised. Referring to any of the most common 8-bit encodings as "extended ASCII" or even just "ASCII" has been very common in my experience.

Yeah. Most people (well, back in the DOS days anyway) who don't know what a "7-bit encoding" is would refer to code pages as "ASCII". I'm sure I did that hundreds of times myself (I didn't know better until I was what, 20?) and I've seen it thousands of times, not exaggerating. Just look at how many "[extended] ASCII games" there are.

I know a lot of DOS people used the term "extended ASCII" to mean specific 8-bit encodings whose first 7 bits were ASCII, but what "extended ASCII" meant depended quite a lot on the context. I came from a different background and it was very clear to me that ASCII was a 7 bit encoding that was almost always stored in an 8 bit byte (I can only think of a few instances in my whole life where I saw bitstreams of ASCII). So that last bit you could do whatever you wanted with.

Having said that, I clearly remember arguing with people that encodings with the eighth bit set were not part of ASCII. As you say, there were many people who didn't understand. My guess is that the GP never interacted with those people. Especially if you were around pre-DOS I think it would be easy to do.

I mention it because I've seen it happen multiple times over my career.

I'm guessing you're just not old enough to have heard anyone refer to extended ascii?

https://en.wikipedia.org/wiki/Extended_ASCII

If you used Macs in the 80s and 90s and shared files with PC users, you heard about it all the time, because high-ascii characters would map incorrectly across the platforms.

The article does specifically limit itself to ' ', '\n' and '\r' and does not say "all whitespace."

Do you mean to say that TFA makes the incorrect assumption that UTF-8 is a single byte encoding?

No, the article states the method will correctly remove the mentioned whitespace characters even from UTF-8 text, because they are part of the 7 bit subset this method works on.

One neat thing about UTF-8 is that all bytes that compose non-ASCII characters will have the high bit set. ASCII values range from 0-127, so no ASCII values will ever have the high bit set. Therefore, a byte that corresponds to an ASCII value is guaranteed to be the same character as the corresponding ASCII encoding.

I think because of UTF-8 most programmers know that ASCII is 7-bit. What you call '8-bit ASCII' is more commonly called Latin-1.

Latin-1 is region specifc. Windows Code Pages and ISO-8859 (depending on the era) would be better terms to use in your example.

What do you mean it's region specific? I don't think that is right. Here is the list of Latin-1 characters (as part of Unicode):

https://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_bl...

There's nothing region specific there.

Also ISO-8859-1 is Latin-1:

> ISO 8859-1 encodes what it refers to as "Latin alphabet no. 1," consisting of 191 characters from the Latin script.

Now there are a few code points in ISO 8859-1 that are undefined, whereas they are defined in Unicode Latin-1, and Windows-1252, but they're mostly the same. The major difference is € and the TM symbol.

> What do you mean it's region specific?

Exactly as it sounds. The characters encoded in Latin-1 are specific for Western Europe and thus may not appear in other ISO-8859 character sets.

> I don't think that is right. Here is the list of Latin-1 characters (as part of Unicode)...There's nothing region specific there.

Unicode is a different set of character sets (note: Unicode isn't even 1 specific character set!) yet again. Latin-1 is not unicode. In fact the point of Unicode was to address the problems that arose with region specific character sets like Latin-1. Hence why there's Latin-1 characters included in Unicode as well as characters from of locales. What you're referencing is the Latin-1 block within the UTF-8 character set.

> Also ISO-8859-1 is Latin-1

It is. But I was referencing ISO-8859 (without the -1) which covers Latin-1 as well as a bunch of other locales.

> Now there are a few code points in ISO 8859-1 that are undefined, whereas they are defined in Unicode Latin-1, and Windows-1252, but they're mostly the same. The major difference is € and the TM symbol.

You're drifting all over the place there:

1. there's no such thing as "Unicode Latin-1". They're different character sets albeit Unicode will have a Latin-1 block (much like Latin-1 has an ASCII block).

2. With regards to your point about the € and TM differences: that is precisely the reason I suggested using ISO-8859 (without the -1) as a reference rather than a region specific character set.

Reply on news.ycombinator.com