Comment by BiteCode_dev | Hacker News Reader

Comment by BiteCode_dev | Hacker News Reader

Parent Full thread

Like anything new, people had a hard time with it at the beginning.

I remember that I got a home assignment in an interview for a PHP job. The person evaluating my code said I should not have used UTF8, which causes "compatibility problems". At the time, I didn't know better, and I answered that no, it was explicitly created to solve compatibility problems, and that they just didn't understand how to deal with encoding properly.

Needing-less to say, I didn't get the job :)

Same with Python 2 code. So many people, when migrating to Python 3, suddenly though python 3 encoding management was broken, since it was raising so many UnicodeDecodingError.

Only much later people realize the huge number of programs that couldn't deal with non ASCII characters in file paths, html attributes or user names, because they just implicitly assume ASCII. "My code used to work fine", they said. But it worked fine on their machine, set to an english locale, tested only using ascii plain text files on their ascii named directories with their ascii last name.

That's in general a problem with dynamic languages with weak type systems. How "Your code runs without crashing" is really really != "your code works". How do people even manage production python! A bug could be lurking anywhere, undetected until it's actually run. Whereas in a compiled language with a strong type system, "your code compiles" is much closer to "your code is correct".

There are a number of mitigations, so those kind of bugs are quite rare. In our large code base, about 98% of bugs we find are of the "we need to handle another case" variety. Pyflakes quickly finds typos which eliminates most of the rest.

This is the difference between people who embrace static typing and everyone else. A static type lover hears that 98% of your bugs are of the "we need to handle another case" variety and says, "well, that means you could have gotten rid of 98% of your bugs with better typing".

No, what I mean is that an additional key comes in (with the json or similar hash) and we now need to do some thing with it, or something different than we thought we were supposed to with it. Typing is not going to fix it because the full cases were unknown at development time.

I don't think a type system can help you with decoding a file with the wrong charset.

The compiler can raise a warning about bad characters. It can't detect all problems, but it certainly can help with some.

It's not the type system, but many dynamic languages are interpreted instead of compiled.

Those bad characters are fed into the program long long after the compiler has done its job.

Not necessarily? I'm talking about constant string literals in source code which can be validated at compile time.

A type system can refuse to turn a `bytes` into a `utf8str` until it's been appropriately parsed.

(It doesn't even need to be a very good or strongly-enforced type system - Go makes it dangerously easy to convert between `[]byte` and `string` by other-type-system standards, and yet everything works pretty well. It's enough to hitch your thinking and make you realize you need another step.)

How does the type system knows the bytes are utf8 and not CP850?

The same way you can parse a string into an integer and it knows you got an either[integer, err].

So it doesn't, and it's not about the type system. Any decode operation in python 3 raise an exception or return string.

It's about the runtime error handling making it mandatory to deal with the error and won't let you panic at run time for this specific error.

I'm sorry you don't see exceptions as part of a type system yet.

> "your code compiles" is much closer to "your code is correct".

That is so far from the truth.

How is it anything but the truth? The express purpose of static analysis, like a type system, is to catch bugs before running your code. That pretty clearly means that code that successfully compiles is closer to being correct than code that doesn't.

I mean, correct as in logically correct and that's just a different matter than being grammatically correct.

The parser assures your code is grammatically correct; the type system assures your code is semantically consistent, which is usually a much stronger guarantee, and by most practical measures will be closer - often much closer, and for total functions on total types, sometimes all the way - to "logically correct".

But this is not a matter of UTF-8 in the code, rather a matter of UTF-8 in the input or output. How does compiling a program ensure that it is robust on a range of inputs?

> How does compiling a program ensure that it is robust on a range of inputs?

This is quite literally the job of a type system: to impose a semantic interpretation on sequences of "raw bits" and let you specify legal (and only legal) operations in terms of the semantic interpretation rather than the bits.

Python 3 encoding management was broken, because it tried to impose Unicode semantics on things that were actually byte streams. For anyone actually correctly handling encodings in Python 2 it was awful because suddenly the language runtime was hiding half the data you needed.

Nowadays, passing bytes to any os function returns bytes objects, not unicode. You'll get string if you pass string objects though, and they will be using utf8 surrogate escaping.

And nowadays, lots of people left the Python ecosystem completely because the 3 upgrade was broken and raising so many UnicodeDecodingErrors for so long. I'm glad it's fixed but it cost too much.

(And it had nothing to do with UTF-8. Actually, it's at least partially caused by the CPython developers avoiding UTF-8 for poor reasons.)

I'd argue that the people correctly handling encodings in Python 2 were vastly outnumbered by the people that weren't, but were getting away with it because the code didn't outright crash. Now in Python 3 it crashes, which is a pain in the short term but better in the long run.

I don't really think it's better - certainly it was not better until many years after release when they started admitting their big mistakes.

But even today, codepoint indexing is too easy and doesn't crash so lots of code is still subtly wrong. Memory usage of a string grows 3x if you add a single emoji. Most libraries are still agnostic about whether they take bytes or str so you still get exceptions thrown with no easy solution. (The growing popularity of type hints is fixing this, but that's not really related to Python 3.)

My Slack name at work is "Τĥιs ñåmè įß ą váĺîδ POSIX paτĥ". My hope is that it serves as an amusing reminder to consider things like spaces and non-ASCII characters.

One of my friends is these days a colleague, with an utterly ordinary English name but his identity management data is full of spurious accents to check APIs do the Right Thing™.

I was delighted recently to stumble on a history of modern women course HIST1158 named Liberté Egalité Beyoncé and I immediately thought two things: 1. Why are our Computer Science courses given unimaginative names? and 2. What a useful test input, I bet some of our systems don't work correctly for this input even though an acute accent is hardly a bleeding edge feature.

I haven't been able to interest any Computer Science professors in fun names for their courses, but I was able in my test environment to name a COMP series course "Untitled Course Name" with a description explaining that "It is a lovely day in the village and there are only two hard problems in Computer Science".

>only two hard problems in Computer Science".

You mean cache invalidation and naming things? :)

And off-by-one errors.

2 just means that the counter overflowed.

Much too trivial. Not even an RTL marker? How about some of the more exotic blank steps? There are some interesting combining characters too!

Reply on news.ycombinator.com