Hacker News Reader

> in practice annotations make code LESS readible, not more

Definitely not true, unless you are writing c++ in Python

> so in practice your team will be skipping annotations - e.g using Dict[str, Any], which defeats the whole purpose of annotations

Using “Dict[str, Any]” doesn’t defeat the whole point of using annotations at all, and adds useful documentation for callers and IDEs. It can also be gradually improved later.

> Typings are still in 0.x version and are constantly changing

Typing is part of the standard library. Some new things are added, but the interface is very stable. Mypy is constantly changing.

> These are solved with proper structure, conventions, testing and developer prep (readme + pairing).

And with some degree of type checking/type annotations.

> If you believe bigger projects need annotations tell me why Sentry didn't go with them yet?

What an odd point to make.

>> Using “Dict[str, Any]” doesn’t defeat the whole point

I think you’ve misunderstood the parent.

That makes two of us. What was the parent trying to say, then?

Python allows gradual typing everywhere. Adding “Dict[str, Any]” gives you value and let’s you catch bugs. Not as much as a more specific signature, but far more than having no annotations.

> Adding “Dict[str, Any]” gives you value and let’s you catch bugs

What types of bugs are you speaking of? Can you share an example?

Remember about constructive feedback method - what, why and how.

You’re asking how type annotations can help you catch bugs. I’m not particularly inclined to synthesise a bunch of examples (especially after the highly condescending initial wording of your comment), it seems fairly obvious and I’m sure you’re smart enough to figure it out.

Can you show me this "initial highly condescending wording of your comment"?

I think we’ll leave this thread here, but if in the future you need to ask something from someone in good faith I would work on refining your wording a bit.

> I’m sure you’re smart enough to figure it out

You are making it personal (well, actually trying to insult), where I was asking you to provide more value.

The classic bugs are passing the wrong type of argument and calling a method on None.

PyCharm did static analysis and caught a lot of these, but it wasn't always that obvious. With a type hint, it's easy for the IDE to spot things. It's easy to integrate mypy into the CI process, etc.

We inherited a py2.7 codebase, written by one dev. It uses a library written also by that dev. We had absolutely no idea what worked how. Sure, looking at the code gave us a very rough big picture, but adding typing helps with the details.

> We inherited a py2.7 codebase, written by one dev. It uses a library written also by that dev. We had absolutely no idea what worked how. Sure, looking at the code gave us a very rough big picture, but adding typing helps with the details.

I see how this could be helpful for legacy codebase - add typings, run mypy and see if it breaks. Good point - thanks.

How many lines of code that inherited codebase had btw?

Quite small in LOCs less than 10K. (A horribly non-idiomatic custom ETL lib, and a big ball of MVC stuff using pyramid and sqlalchemy/zope.transaction.)

e.g. say you're writing some code that does something like this:

account = queryDatabase(email)

updatedUser = update(account)

ping(account)

queryDatabase returns a dictionary. Later on, someone decides that a single username can have multiple accounts, so someone changes queryDatabase to return a list of dictionaries instead of a single dictionary. Now the above code is broken, since update expects a single dictionary. Of course, you could grep through the whole codebase to try and find all the problems, but it would be easier and less prone to human error to rely on the type system to fail, especially if the change was something less contrived, and if changes like this were happening all the time.

The effect of this could be limited by other means (e.g. naming the function better) but type systems are just another way of adding robustness to a program.

I see your points and I was going through pretty much all of them when deciding whether to go with typings or not (eventyally we did because I wanted to try it out and because pressure from Java people who found it odd that we don't care about types - actually we do, but instead of types we have strict naming conventions).

The point with sentry was that when making a decision in field you don't have much experience in, you look what other guys with resources are doing since they had these conversations already.

IMO in practice you will realize that in most cases typings are more of a burden than help - devs don't know typings and will spend a lot of time thinking how to annotate, making constant decisions whether to annotate/not annotate (aka incremental typing). That inconsistency drives me nuts - aka defeats the whole purpose - you do something right or don't do it at all right?

I see the point with documentation, yes it is nice to have, but more important like I said before is proper onboarding readme file and docs + willingness of other devs who are working on the codebase to handhold when necessary.

> you do something right or don’t do it at all

Down that path lies madness. The Boy Scouts rule is to leave the camping ground cleaner than you found it. It doesn’t say the campground has to be spotless.

Great analogy.

I see how this sort of attitude can be seen as mad/neurotic. However the good side is that it prevents adding more complexity (rules for when to annotate/not annotate is adding complexity).

> and adds useful documentation for callers and IDEs.

THIS. This is why I started using them, along with standardizing my docstrings. Enormously helpful when you're trying to remember what the function is expecting to just hover it and find out.

Yes, they are probably most useful to me for the IDE completions and docs as well. Raymond Hettinger gave at least one talk [1] where he discussed how there are tradeoffs: reasons why you might use them and reasons why you might not. At the end of the day you can pass whatever you want to a function and the only reason to use types if they make your code better—but there are other tools that might be more worth your time in some situations.

[1]: https://www.youtube.com/watch?v=ARKbfWk4Xyw

I recently converted a medium size project from Python 2 to 3 by first adding type annotations to everything, then getting mypy to pass with both versions of Python and then fixing up the remaining issues on the unittests. The functionality of the code depends a lot on external services that are accessed via HTTP/subprocesses. This makes it pretty hard to test well; at best your tests end up exercising an inaccurate partial facsilmile of the external service.

Overall I didn't really love mypy; the type system it implements seems relatively simplistic and it was frequently easier to change the code rather than work out how to write working annotations for the existing code. The lack of support for decorated property setters in particular was a big problem; this particular codebase makes extensive use of those to annotate methods that require mutable access to the underlying data. Nevertheless I'm convinced that this approach to the migration was much faster and less painful than it would have been to try and figure out all the string/bytes differences without any tooling support except for the tests. And post migration the code seems overall somewhat higher quality than at the start.

On balance, I think the lesson was actually that Python isn't a great choice for this kind of project. When you are depending on external services the ability to unit test everything is greatly diminished, but the ability to make type assertions is not. So for correctness you want the latter to do as much lifting as possible. That suggests that other langauges such as typescript (if you still want a language in vaugely the same category as Python), or Rust (if you don't), which come with more expressive, better integrated, type systems, might be superior options.

What version of Py3 were you using? There have been a lot of additions to type annotations in recent releases that have made them better.

I'm not sure what you mean by lack of support for decorated property setters? Can't you annotate them like any other method?

Mypy doesn't like function attributes, https://github.com/python/mypy/issues/2087

> you will be better off investing into proper testing (I mean here code coverage)

Type safety and tests are completely orthogonal issues. Gary Bernhardt explains this really well:

https://www.destroyallsoftware.com/talks/ideology

That is really on point, thanks!

ps. watched the presentation - the idea here is that static type checking can eliminate entire category of possibile bugs which are impossible to be eliminated by test - for example enforce variables to be of specific type only.

It doesn't work that well where in the most bug prone places typings are made optional.

I'm actually quite curious as to where annotations make things less readable. Do you have any examples? I can imagine if the annotations are wrong and somehow pass checks that this would be true - which is a possibility, I haven't worked with type annotations in Python at all.

In our case biggest problem arose with Pandas, where each dataframe transform would need to have it's own type. When there are a lot of transforms it is more readable to infer what type variable has just by looking at transforms (more or less) or when needing to be precise step through the code with ipdb/jupyter etc.

What about incremental typing? That's what we did, and to be honest I wouldn't do it again. Yes, it's nice to see what type function is returning/arguments it's accepting, but if the name is shit type won't tell you much - you are better off spending more time of figure out better naming convention (prepend dataframes with df_ for example) and keep at it.

Another problem with incremental typing is that now of course you will have to decide whether to annotate/not annotate, and when reading code thinking why this is annotated and that is not annotated.

Right. Similar problem to Typescript - generally its best users are those already sold on static types, for many others the temptation of <any> and friends is too much, and interacting with untyped/poorly typed libraries + poor type inference can be really frustrating. Although in that case it's still a massive gain for new projects, in my experience.

another solution yet is going back to one of the fundamental principles of message based OO, leaving interpretation of what objects receive and how to handle them to the object.

I've always disliked the idea of using static code-evaluation tools on dynamic object oriented languages, it kind of defeats the point, which is runtime dynamism.

That's not the solution, it's the problem. A large system is impossible to understand when state could be anywhere and any part could be coupled to any other part.

I think you're partly right, but ultimately runtime dynamism has failed; even in Python, the costs of being truly dynamic are higher than the benefits.

the intention of that proposed solution is radical decoupling though. Think of objects being used analog to network servers. The internet doesn't go down because someone's server changes (the BEAM VM probably comes closest as far as languages are concerned).

Objects communicate messages and status and when something crashes it's up to the object to do something, often restart, but either way it's a matter of interpretation.

It's the static typing that couples because it introduces global behaviour through its types, which impose meaning from the top. In a truly dynamic system, nobody needs to understand the entire system.

I admit it's a fairly theoritcal point because I don't think many people even embrace this paradigm any more, and ironically if they do it's in functional languages. But to say it's failed I think is not right, because nowadays people write dynamic languages basically like they'd write Java, they don't embrace the dynamic aspects.

> It's the static typing that couples because it introduces global behaviour through its types, which impose meaning from the top.

I don't think that's true; static typing on a local scale or at interface boundaries is useful precisely because it enables local reasoning. Without those constraints you have no chance of understanding any part of the system without understanding the whole thing. If you go so far as to expose all state manipulation in the type system then that becomes sort of global, but even then I'd argue what you're actually doing is not introducing global coupling but rather exposing the global coupling that was already there.

> But to say it's failed I think is not right, because nowadays people write dynamic languages basically like they'd write Java, they don't embrace the dynamic aspects.

I think that very fact is the proof that it has failed; even in languages that were designed for it, people prefer not to follow that paradigm.

My personal anectodal experience :

3 weeks ago, I had a new pet project, and decided to go with Python again, which I left many years ago, before Python typing was a thing.

I didn't know where to start, so I started by creating a file with data models I would need. Basically C struct-like / record models. All the classes inherited NamedTuple, and I typed all the attributes. I then created a few pure functions based on those classes. NewType is a great addition. When you work with many int or float variables representing different things in real life (price, amount, quantities, etc), it makes the code cleaner and catch some mistakes when you are tired at the end of the day.

I gradually stopped to type things when I introduced pandas into the project and types started to be way more abstract. There is probably a way to make typing work with pandas, but I wanted to learn pandas first and get the project running, so I didn't really try. I expected pandas to do dynamic things under the hood to work, so I didn't bother to understand how types work with it.

Of course, being a hobby project, there is no unit test and no documentation, so I may not see all the pros and cons of typing. But I have a feeling gradual typing is an interesting idea that will continue to improve.

It's nice to have the support of static typing when you need to, but also be able to "just hack" a solution real quick when needed.

> there is probably a way to make typing work with pandas

So we did use a lot of pandas as well. The way to make it work is to create custom types and of course each dataframe will have it's own type which is going to be a total mess.

typing pandas seems like an r&d-level problem as it seems quite close to tricky areas like row types / dependent types that mostly only work in theory

attempting manual workarounds like pure nominal typing for it as you describe is against the grain of a type system and pushes a lot of work to the user. We just... don't. We stick with class ('pd DataFrame'), and the one extension we are thinking is Index, while getting into actual columns gets into a mess quite quickly.

I tried to engage some PL researchers on this awhile back but no go. IMO would be a great project for most type system research students.

Not exactly this, but I've had some success using logic programming (minikanren) to statically assert certain facts about dataframes in spark. E.g. "if Row_a is filtered to Row_b, and Row_a is not null, Row_b is also not null".

> in practice annotations make code LESS readible, not more

Agree, and disagree. I think it depends on lots of things, including how "heavy" you are with type annotations - I think annotating everything, or mandating to, including inline variable definitions - is over-the-top and usually makes the code harder to read. Likewise, chasing down every minor type error for the corner cases can end up wasting loads of time for almost no gain.

I also won't argue that sometimes the type definitions are excessively verbose and get in the way of reading code, and your point about less experienced devs is also true - for developers who aren't so familiar with type annotations, at worst, they make it harder to understand.

But carefully used, they can really aid understanding a function or codebase. We have a large, somewhat messy codebase open source academic codebase where the input to a function could come from dynamic dispatch some ~5 layers away, and usually the first task of diagnosing a problem is working out what you've actually been passed. Type annotations help solve this problem, as a somewhat statically verifiable documentation of inputs. We've found bugs using them.

It's probably not controversial to say that expected types/inputs aren't always obvious from the context of a function, and so need to be documented, to help people using the code. The alternatives to annotations are docstring documenting of types, which is defining types but decoupled and in a non-statically-checkable way - so easy to get them out-of-sync, which is easy when developers are undisciplined; naming (hard, and messy for compound types), pyi files (checkable, keeps them out of the way, but decouples definitions from code), or type comment - which solves some of the readability and checkability, but is still harder to keep up-to-date than inline.

So, overall I like them and have seen them bring value to the code I work with, even if that's just documentation, but don't think it's a black-and-white issue. I've certainly eschewed using them when helping less experienced developers write small personal scripts, but use them in my own code or when refactoring something vaguely defined to cross-check my assumptions.

> remember most Python devs have little experience with staticly typed langauges, so in practice your team will be skipping annotations - e.g using Dict[str, Any], which defeats the whole purpose of annotations

Using mypy or pylance gives benefits even without annotations, and while you can often do better than Dict[str, Any], rough cut annotations like that often, IME, provide value. (For one, they’ll catch paths that would allow either the dict or a key in it to be None when that isn't expected.)

Are there statically typed languages that do support reloading parts of the codebase?

Type hints in an interpreted language just don't seem to make much sense to me, as in, why use a dynamically typed language when you actually want a statically typed one? What are the benefits or staying in between the two?

Besides of course a legacy code base...

> If you believe bigger projects need annotations tell me why Sentry didn't go with them yet?

Because we just came off of Python 2 and typing under Python 2 is awful.