Hacker News Reader

Comment by bhaak | Hacker News Reader

> "If you know what UTF-8 is (and you should) every ASCII file is correct UTF-8 as well." > > I mean kind of? ASCII is a character set, it doesn't define the on-disk representation.

Excuse me for being pedantic as well but UTF-8 also doesn't define the on-disk representation and is just a character encoding of the Unicode character set. But they define bit representation and for that the statement is true.

> An ASCII file saved with each character packed into 7 bits isn't going to be valid UTF-8 without some manipulation. This is just an odd claim to make given the earlier discussion of varied word lengths and transfer formats that weren't 8-bit clean.

How these character sets are transported over other protocols is outside of the scope of their definitions.

View on HN

ASCII defines a 7-bit character set. It's modern convention to encode that with each character as an 8-bit value with the top bit clear, but there's no inherent reason for ASCII to be represented that way, and a single counterexample disproves the assertion that all ASCII files are valid UTF-8

PRIMOS[0] serves as a counterexample; it used ASCII with the top bit set.

[0] https://en.wikipedia.org/wiki/PRIMOS#Data_access

It’s funny how you describe it as “ASCII with the top bit set”. As in, by default ASCII doesn’t have the top bit set. You can have malformed Unicode too. There are plenty of invalid ranges defined in the UTF-8 spec, does that mean that UTF-8 is just a convention and those malformed pieces are counterexamples to the convention?

>They developed hardware optimized to run FORTRAN, including machine instructions that directly implemented FORTRAN's distinctive 3-way branch operation [=Arithmetic IF]

Hahaha oh wow. That's amazing. Fun fact:

Arithmetic IF makes handling exit codes significantly less annoying¹, and that's because Arithmetic IF is how we came to get signed integers as exit codes in the first place.

¹at least in theory, in practice half the people implement the codes incorrectly because they don't realize this, and so…

Holy cow. Never in my wildest dreams did I ever think I'd ever hear of PRIMOS again.

Apple II as well. When displaying text, bit 7 determined if the text was flashing (0) or normal (1). (There was also inverse with bit 7 and bit 6 clear). Since bit 7 had to be set to display normally, some text editors saved their ASCII text files with bit 7 set.

No that’s not a modern convention. You could argue that any higher order bit beyond the 7th bit is undefined by the specification but the order if the bits is clearly defined.

Anything with a defined 8th bit would be an extension of ASCII.

But it was obvious back then that if you use 8 bits for a character that the 8th bit would have to be unset unless some further processing was happening.

Even more so in the context of UTF-8. 8 bits bytes had won by then and in that environment ASCII was a thorough sub set of UTF-8.

Elsewhere in the thread is an example of a system setting the top bit when storing ASCII in 8-bit files. But storing it as an 8-bit encoding is also a convention rather than a requirement, and any file with an alternative encoding would not be valid UTF-8.

“Modern convention” that is defined in RFC 20 (https://datatracker.ietf.org/doc/html/rfc20.html), from 1969 (referencing the ASCII standard from 1968).

Something I noticed in that document (unrelated to the topic of the RFC) was an alternative way of listing hex values as decimal numbers separated by "/". I really like that (I always struggle with converting A to F in my head). I guess it didn't catch on.

> ASCII X'2E' or 2/14

> X'OD' or 0/13

Ye. In my head 0x10 looks smaller than 0x0B since 0xB is bigger than 10.

Reply on news.ycombinator.com