Comment by ReflectedImage | Hacker News Reader

Comment by ReflectedImage | Hacker News Reader

Parent Full thread

I don't think Numpy is mostly written in Python. It's C.

So using type hints to annotate your flask api, etc. to generate docs is great. No problem.

The issue is that there have been a lot of Java developers switching to Python and they want to pretend that Python is a statically typed language (and use tools like MyPy) because it's more familar to them.

Python is not a statically typed language and treating it as one leads to terrible results. Especially in large projects.

The issue of course if that the Java developers have no point of reference on what a successful large Python project looks like. So they think they are doing great with MyPy when if you compare what they are outputting to a proper large Python project, it's pretty clear they are doing terribly.

When you use Python properly you write self contained microservices. Typing information exists but only on the external interfaces e.g. API. You also don't share business logic code between the self contained microservices because that massively decreases the maintainabily of the overall system. You show your Python code is correct by using Unit Testing and Mocking.

Basically, there is a way to do large Python projects and MyPy (and other static type checking tools) have no place in that story. It only exists to support people in making bad decisions on their codebase.

Numpy has a large amount of python code. C is definitely used in places, but majority of numpy is python. Not all operations need C implementations especially as many can be built on top of other python functions which do wrap C. Tensorflow/pytorch similarly have a large amount of python code. I mostly work on developing a library that builds on tensorflow and is like 50K lines of code. Dividing that library into services makes little sense. One of the recent typing new features is mostly devoted to numerical ecosystem in python (variadic generics to do template like types for matrices). I don't want to use dynamically typed language, but python is clear leader in ML ecosystem. A lot of my department does work that builds on ML research/libraries and those are mostly found in python.

If other languages had comparable ML ecosystems then a different language may have been chosen. But at moment today there's no competitor anywhere close. One lazy metric, I would estimate 90%+ of research papers to ML conferences are in python.

"Dividing that library into services makes little sense." No, but dividing the library into smaller sub-libraries, that only interact with each other over well-documented interfaces, does make sense. It's called encapulation.

If at some point you decide the way one of those sub-libraries works is wrong, could be done better, etc. then you can write a new sub-library that just provides the same public interface.

One of the reasons why Python is so successful is it's usage of dynamic typing. Obviously, if you lose static typing some else needs to take it's place, in Python's case, stronger encapulation.

> If at some point you decide the way one of those sub-libraries works is wrong, could be done better, etc. then you can write a new sub-library that just provides the same public interface.

That's...that's literally how type systems and interfaces work. You do know Python supports behavioral subtyping (Protocol), right?

It sounds like you certainly have had some bad experiences with poorly-written typed python. But that speaks more to those maintainers not knowing how to actually use types effectively, vs a shortcoming of static typing in python. Python's type system has plenty of shortcomings, but has plenty of escape hatches as well.

The size of the enclosed section is much bigger. A large section of code has an small well-defined interface.

What you get with typing and interfaces is that all the code has an interface. Small sections of code have a large badly-defined interface.

The main reason people are using typing in Python is to support large Monorepos. Because everything is typed in them, all the code in the repository depends on all the other code in a spaghete dependency graph.

This results in the following issues:

* Small code changes lead to hour long unit test runs since most of the unit tests need to be rerun for every change.

* It's impossible to update the Python version since all the Python code needs to be updated at the same time.

* Long check-in times for changes running MyPy over 100k lines of code.

* Maintainability issues since code can't be updated without knock on effects all over the codebase.

Okay, so what are the benefits of using types in Python:

* On average MyPy type checking will catch 1 bug per developer per year, which wouldn't of being caught normally.

* It makes people who previously coded in statically typed languages feel more familar with the code.

Basically, if you look at the Pros vs Cons, you should ditch the typing checking. It's a net negative to the codebase.

> Small code changes lead to hour long unit test runs since most of the unit tests need to be rerun for every change.

You're writing unit tests wrong if they are taking that long. Unit tests should be quick.

> It's impossible to update the Python version since all the Python code needs to be updated at the same time.

I've done it, so, objectively not impossible.

> Long check-in times for changes running MyPy over 100k lines of code.

TFA addresses this. It can be greatly sped up with use of caching in CI.

> Maintainability issues since code can't be updated without knock on effects all over the codebase.

That's exactly why static types are better - automatic refactors.

> On average MyPy type checking will catch 1 bug per developer per year, which wouldn't of being caught normally.

I catch several per day, simply from IDE highlighting, in real time, and fix them immediately, which only works because of the type system. (maybe it's wrong to call these bugs at this point, it's more like proto-bugs which never even get checked in cause there is immediate feedback)

> It makes people who previously coded in statically typed languages feel more familar with the code.

I came from a fully-dynamic-everything python world, and types were a breath of fresh air.

You've clearly been burned by (a) bad codebase(s), but I think you are drawing all the wrong conclusions. All the benefits of narrow interfaces, easy refactors, high maintainability, can be had with static typed python. I will admit it's much more challenging to be really competent at it than it ought to be, but this has been improving rapidly.

Encapsulation is a concept that is orthogonal to whether or not you use typing annotations in a language. You're conflating two completely different things and wrongly assuming that typed code is ipso facto poorly encapsulated. As most of your arguments stem from this premise, I find them very weak.

Reply on news.ycombinator.com