Python’s “type hints” are a bit of a disappointment to me
uninformativ.de
uninformativ.de
My editor now has a fairly deep understanding of my code, and can tell me of all sorts of surprising errors I'm making before I run the code. There have been a few times where I found an error, went into the editor and saw I had missed a error message about that line. Shout out: Using LunarVim with LSP and TreeSitter.
The other thing I'm enjoying is that libraries can use them to make things happen more automatically. I believe it was Typer (the CLI argument parsing library) where if you declare an argument to a function as "files: List[Path]", it understands that the argument will take one or more files, or "-" to mean stdin. If you just say "file: Path", it understands it is a singular file.
I was curious about type hints when they first came out, wasn't really expecting to use them but they seemed cool, but now I'm totally sold.
I agree this is useful, but to me the most useful feature of types are the self-documentation. It's crazy the difference between a non-typed library with shitty documentation vs a typed library with shitty documentation.
If the author / last person who touched the code misunderstood it then there is a 100% chance there is a bug here.
Knowing that off the bat is quite helpful. And it also tells something about the code quality if it is a common occurrence.
Which need would send me back to pypi to look at other prior art, if possible.
Hopefully your CI does?
Had I had MyPy back when I was working in data science, I would've saved countless hours and headaches.
No, the code in data science isn't bad because of the lack of typing. The code is "bad" mostly because those writing it are relatively fresh from starting to program. Also there is more pressure to make things possible, often just to run it once, and neglect repeatability or scaling to larger code bases. Different emphasis. That doesn't mean an experienced full stack developer would do Data Science better, because he might lack a lot of skills that matter more in that domain.
I'm not sure if that's true. Doesn't Pandas handle ETL and some anaysis? There is nothing inherent to ETL that makes it a hard problem with compiled languages.
In your opinion, what does Pandas do that's hard to do with compile-time languages?
This even starts with basic Numpy and handling tensor objects. It's not easy for a type checker to understand what operations you can do with what shape of tensor. Worse, most often you don't know (or want to know) some of the dimensions or even dimensionality of some of the objects. Then it is impossible to check all of this at compile time.
That doesn't sound like a Python problem.
Instead, it sounds like the natural consequence of numpy being designed in a way where their data types aren't organized into subtypes, and leave that as runtime properties. This is a natural reflection of numpy's take on vectors, matrices, and tensors, which in terms of types are just big arrays with runtime properties.
To put things in perspective, in C++, Eigen supports static dense vectors and matrices whose size is specified and known at compile-time. I'm sure Python doesn't impose addition static type constraints than C++.
Significant types in ETL-style applications typically come from outside (e.g. a certain CSV column in the input file contains a date in YYYYMMDD format, or maybe YYYYDDMM, figure it out, and don't forget time zones or your accounting will go wrong).
Then types are mostly complex but obvious and easily deducted (e.g. multiplying matrices of compatible shapes necessarily gives a matrix of a certain shape, why should the program say anything more detailed or lower-level than "do a matrix multiplication" or "do a tensor product"?); they are an often dynamic and unpredictable property of the data, not a useful abstraction.
What, mutate data with known structure? What exactly do you imagine is hard about this, let alone notoriously so?
A pandas dataframe types to something like Iterable[SomeKindOfProductType[int, str, str, ... (78 other columns)]]. The formal type of a dataframe in the middle of a transformation is... not very useful to know.
The number and types of the columns aren't necessarily known at compile time. Which leads to runtime errors. Even in a statically typed language, such dataframes are a kind of "dynamic typing escape hatch". As complexity of a software increases, such mechanismus of dynamic typing and throwing runtime errors creep up all over the place.
It's certainly very useful to know, even if that knowledge is indirect via the type of the resulting dataframe after all the transforms.
This resonates with my experience. I had the opportunity to work on a DS codebase written entirely in Scala with all the typing, parallelism, actor model, whatnot. Basically I joined the company because of this technical factor. It was fun until I figured out that DS was "typed IF-THEN-ELSE written by Java devs in Scala returning stuff the users complain about with high reliability within milliseconds". Now I am happy to be back to the single threaded untyped Python world. Still no bugs in production, because we validate all requests to death, have unit tests and integration tests running on real data not on mokups. Basically we follow the principle: if the integration test passes, our typing is just right, or at least good enough. Funnily all the typing errors we catch are caused by wrongly typed data, coming from the productive system written in a typed programming language... what a strange world.
I can see you are enjoying the life outside of a highly regulated industry. Having certain kinds of production data in tests (or feeding that to test environment) would be a major audit finding in any finance or healthcare company.
Makes for both a blessing and a curse.
...thereby destroying the production-like features for which you want it for testing, which you then need to recreate and reintroduce, so you might as well just synthesize test data in the first place, since that's what you end up doing anyway, in effect.
No, in actual practice you don't scrub the stuff you actually need to test.
No, I’ve spent ~20 years in healthcare, with this issue as a frequently recurring issue.
> No, in actual practice you don't scrub the stuff you actually need to test.
In actual practice, the stuff you really need to test often overlaps with the stuff minimally required to scrub to legally deanonymize the data. The most common scenario I’ve seen trying to do this is both creating most of the work of generating synthetic data and failing to legally deidentify the source data.
This is the largest difference. When there's no expectation of code lasting beyond a very short lifespan, why go through the effort to future proof things, improve maintainability, have better ergonomics, etc?
Sounds a lot like F# type providers to me.
https://thesharperdev.com/introduction-to-fsharp-type-provid...
And good luck teaching beginning scientists F#.
The output of most analytics is a report. For longer lived processes it seems easier to just hand off
Static typing is completely viable and I hear those arguments mostly from those who didn't use Python for a long time. The added productivity makes up for a little more debugging while the program is running.
I'd also posit that many Python codebases are somewhat less "untyped" than one might assume. Django for example does a lot in its ORM and Form classes to make sure the right stuff goes in the right slot.
The added productivity of non-typed Python is such a ridiculous myth to anyone who has to maintain significant Python code bases. Sure, it makes you more productive for one-off exercises but the moment you’re having someone else inspect your code the productivity gains vanish. This largely true of dynamic languages more generally except in niche caches (e.g., ORMs) where types are strongly implicit.
I say all this as a mathematician that is used to reading papers full of symbols packed with implicit information. The code I’ve read from gung-ho dynamic typing advocates is some of the most painful drivel I’ve ever had to wade through.
I can compare Python's productivity to other languages, and it beats all of them so far. Another lesson I learned from working with statically typed languages: The quality of the code is more dependent on who writes it than on the language or even the ecosystem. But people are productive much sooner in Python than they are in C# or F#, and then we can work on the foot-gunning.
Python's productivity isn't a myth. This is proven by its popularity and by successful projects across academia and industry alike.
There's also the argument that one is most productive in what they know. As someone who has spent far too long with Haskell and OCaml, Python's semantics and libraries often seem surprising and counterintuitive to me.
I know that Typescript helped me over Javascript. I don't get that benefit from Mypy yet, but the "pain" just isn't as great as it was with JS. And I've seen what inexperienced programmers do when a static typecheckers try to force or encourage them to do the right thing. It's not necessarily the right thing, just a lot more complicated than it needs to be.
No, I have worked at a variety of large institutions that have millions of lines of Python. I work with it often. The use of Python is more about it being accessible for non-developers, which is why they also teach it in schools; and the need for a standard. It is the new "Visual Basic". Again, it is not more productive in all situations.
There are a lot of "libraries" people write for their use only, who then put them on Github as a token of good will.
You as a user of someone else's library can either contribute to it or learn to use it. Otherwise, write your own equivalent.
And be judged for wasting time reinventing a wheel?
That is never a waste of time.
I took a random stab in pandas but they’ve adopted type annotations so that defeats the point. So i thought old-school code bases - zope sprang to mind, but that’s not a good example either because they use docstrings and it’s kind of obvious. So i tried gunicorn on a whim and they have even more attention spent on their docstrings.
I just want to sample this feeling of lost confusion in python so that i can figure out how I’d deal with it.
def _new_conn(self):
"""Establish a socket connection and set nodelay settings on it.
:return: New socket connection.
"""
...
try:
conn = connection.create_connection(
(self._dns_host, self.port), self.timeout, **extra_kw
)
...
return conn
What's the type of "conn"? It certainly wasn't clear to me that it's a socket.
But just now I looked at upstream and I notice they've added type annotations, which do greatly improve things:
https://github.com/urllib3/urllib3/blob/e16beb210c03c6f5ce4e...
def _new_conn(self) -> socket.socket:
...
That docstring isn’t enough for the problem you had though, e.g. I wasn’t clear from that if they meant a socket.socket or maybe some special socket derivative type so i hit go to definition and create_connection is a short method - i can see it is indeed a socket.socket being created, configured and returned.
I’m guessing you’re not using any developer tooling, maybe a straight text editor i guess?
This kind of tooling has been around for a long time, i remember using rope and jedi in vim back around the late 2000’s i think. It’s gotten a LOT better since PyCharm appeared mid 2010’s i think - first i recall using it must have been around 2014-ish. These days, it’s even better. I would really strongly encourage adopting some kind of tooling that lets you navigate and has some basic refactoring - rename is usually 80%+ of the value there so it doesn’t need to be complex. A few ergonomics like having the ability to jump to test and call from repl with a keystroke - you’ll turbo charge your comfort and productivity.
That has to be our difference i think because it took one mouse move and one key chord to completely bottom out.
- Static type systems are an effective form of documentation
– Static type systems reduce the effort to fix type errors
– Static type systems may not be helpful in preventing semantic errors
It would have been awesome to have evidence for #3 but even just #1 & #2 could be useful, albeit to a much lesser extent. Could we quantify by how much?
It’d be important to quantify because if static types mean a developer can be done sooner, then they have spare time to spend improving the software. So do these differences with static types translate into benefits for the developer?
The author of that paper says no. He published this other paper just after that one: https://www.ics.uci.edu/~jajones/INF102-S18/readings/23_hane...
Other take home message is that the famous 10x programmer was found in both groups. The one with dynamic language was faster, just as the argument for dynamic typing goes.
The benefits of masks against spreading respiratory viruses are easily and objectively proven. Completely different thing.
I've also programmed extensively in Pascal, assembly, and C. I get it about types. Don't forget C has 8 different types of integers that have to be kept straight. ;-)
On the other hand, I realize that any of my programs is but a little nub of code sitting on top of large and expertly maintained libraries. I'm grateful for whatever the authors of those libraries are doing to keep their sanity, and am not surprised when I see type hints in those codes.
In my opinion there is no excuse not to.
Luckily IDEs can auto add the imports, but sometimes a slightly mistyped piece of code auto adds an import I need to delete, I love avoiding imports and circular imports in PHP. The other nice thing is avoiding random python behaviour differences in MacOS, Windows and Ubuntu. Code that works fine in CircleCI but not locally. Works on a Mac but not on windows etc… I didn’t experience that with PHP.
Not sure there are a huge number of long term PHP devs who switch to Python or the other way around to have a reasonable opinion.
Adding stronger typing, like type hints, could reverse that, of course.
The reason not to is performance and reliability. Typing helps with the latter, Python is just bad at the former.
Not to mention the notorious circular import problem, that is much more likely in a big codebase.
If I have a custom class, that class has to be passed as a type hint for functions. What If I use that class in another module, but have to import it inside this one that has a type-hint only?
Well, let me introduce you to their genious solution:
from _future_ import annotations
from foomod import FooClass
def myfun(foo: FooClass):
return foo.bar()
Not once did I find the additional typing useful in my actual day-to-day work.
In my experience its the other way around. Most people who are a fan of dynamic typing have worked in systems with types. The world is largely coming around to the view that types make more consumable and safe software
If so, I would counter that adding types isn't particularly onerous and adds very little overhead to development time, and over time they'll tend to increase your productivity since you'll make less errors, IDEs can give you better hints, etc.
As to the percent of Python developers who use types / type checking, it seems to me that's been changing somewhat rapidly over the past few years. It's certainly greater than 1%.
Gradual typing is based on effort. Most people won't spend the effort. And they don't need to, most of the time, either.
Each new line or character is a line to maintain, there is a cost associated with that. Tests have costs but are extremely valuable, they are a no brainer to me, types are way less obvious.
I have spent a lot of time maintaining python types. I have no idea how many bugs it has prevented. I am not sure it is a large number.
I know there have been bugs in types, bugs in mypy and dependency changes which created work which only existed because types were in place. There have definitely been days worth of types maintain.
There are the extra type imports, extra time to run tests and integration tests etc.
It is not as straightforward as you might initially assume.
For me, python is mostly a calculator, and it seems faster to try a calculation and immediately see if it runs and/or looks correct, rather than try to implement type hints et al (though I still use type hints occasionally).
List[Tuple[Tuple[str, int, str], Tuple[int, float, float]]]
How about no?
What is not easy is type-hinting it. What if I want it to be both an list and an iterator? Well, I would have to use Union.
I'll leave it to you to write that Union "type".
Union[List[Tuple[Tuple[str, int, str], Tuple[int, float, float]]], Iterator[Tuple[Tuple[str, int, str], Tuple[int, float, float]]]]
Starting from 3.8 (I think), you can also directly use dict[T, U] and list[T] rather than having to import Dict and List.
This as opposed to a legitimate type system where complex types are classes themselves which don't need unraveling. There is no need to alias a type and there is no need to maintain separate complexities for the object and the type.
Type aliases are okay if they're not overly complex, having used them in Python and other languages to good effect, e.g. to avoid duplication or for clarity.
I'd suggest using type annotations where it's not overly onerous--which is probably most places--to get the benefits without much effort but perhaps avoid them or use simplified types where it's too complex.
I also agree that there are places where typing is unnecessary and doesn't add much value, such as in certain scripts.
Like many things in programming, it's all about finding the right trade-offs.
This is exactly the kind of situation where you're supposed to use a named tuple or a data class to make it clear what's what.
Named tuples are just immutable classes with slots. Unless you want to slow your code down considerably, it's best to avoid them.
Thing = List[Tuple[Tuple[str, int, str], Tuple[int, float, float]]]
def foo(t: Thing) -> Thing:
...It's literally a tuple containing two three-tuples inside a list.
Do a for loop over it, unpack it and you are done..
What I'd like to do is integrate something into our pre-commit hooks.
My stack is pycharm, mypy, pydantic, sqlalchemy stub, several mypy plugins, black, isort, tabNine completion, and a bunch of custom scripts/Make to automate the common tasks. I'm not even sure I'm on the same planet as the author. I can practically spot incorrect type hints, mismatched types, weakly defined interfaces, from orbit, that's how well they are highlighted. Since every type is known, between pycharm and tabnine I can type 2-5 keys and spit out entire lines of code. And since it's all typed, automatic refactor is a breeze.
I remember the dark ages, before type hints. It was rough. I lived in the REPL. Write 3 lines, run, Exception. Write 3 more lines, run, NoneType error. Ad nauseum. Now I write for hours on end and don't wince when I hit execute. And I don't have to go bug coworkers with questions like "hey what keys are in this dict when we redirect from the oauth endpoint".
Sure, I have my gripes, but it's mostly with the lack of power of the type system, like lack of higher kinded types. But I'm confident it'll get there.
Come to the typed side. We have less mental fatigue and screaming at neverending runtime oopsies.
Many popular libraries, however, don't have type hints or use them inconsistently (looking at you numpy, pytorch, tensorflow, scipy, ...).
Always great fun to see function signatures that use type hints for the parameters and return Any.
So it really depends what kind of code you write - web frameworks might be fine, data science and ML is definitely not.
Otherwise, don't bother.
Christ. What a mess. With a stack like that, it's a stretch to say you're "writing in Python" anymore.
You silly! Just use these carefully curated list of 28 different tools (that I got to after years of trial and error) and you'll have a somewhat acceptable tooling and type support in Python.
If you really want, you can even throw in a little pre-commit hit hook that runs pyright and prevents you from commuting code with type errors.
First of all, Black and Isort are literally code formatters that have nothing to do with type hints.
Second, the whole point is that you can actually develop powerful dev tooling for Python now, whereas in the past it was difficult or impossible.
> pydantic is primarily a parsing library, not a validation library. Validation is a means to an end: building a model which conforms to the types and constraints provided.
Anyone whose used Typescript or even strong typed languages knew what to expect from Python + mypy. No surprises there. But pydantic's use of type hinting to create a remarkably dense serialisation library, and then fastapi's leveraging that to create REST interfaces along with automagic documentation generation - well that wasn't when I was thinking would happen when I saw the type hinting proposal.
Who cares if I'm "writing in python" at all? I'm solving problems. Honestly a good IDE (pycharm or vscode) will give you 80% of the ergonomics.
Otherwise the exact same stack as yours, we use fast api as well. Also a big fan of using dataclasses everywhere.
I work with large code bases (100K-1M lines) and can't look back to the time we didn't use Type Hints. Would maybe dismiss it only for scripts or very small projects.
This is garbage and moving backwards as a programmer. I loved python when it was easy to use with any editor and with few plugins. Now I'm just going to switch to a statically typed language where I don't have to install a zillion plugins to get first class typing.
Python type-hints bind you to a lot of other sister projects of Python, which makes you question if they are even official addition to the language.
I think that's also what the OP's article tries to convey.
So developers are creating complex custom types with union[custom_type_1, custom_type_2] where each custom type could have more unions of other custom types. These custom types then get imported everywhere. It is utter garbage.
Routes = Union[GooglyRoutes, MetaRoutes]
Where, GooglyRoutes=Union[List[CloudyRoutes], Sequence[AdseyRoutes]]
Where CloudyRoutes = Union[RouteContainers]
And then they'd import these types all over the python repo creating all kinds of import issues. Fml.
def register_route(route): …
Like that unwieldy nested union already exists in your code, adding the type just documents it and the type checker makes you handle all the cases.
Is there something specific about imports that don’t work with type alises?
class Route:
_route = [] # can be GoogleyRoute or MetaRoute
But we can't do this with the spaghetti that python type annotations are.
Also, I don't know the grip with productivity tools.
People not using said "modern" IDEs are just stuck out of using the language. Is this progress? Even with modern IDEs, there is a difference between what different IDEs do. So now the language is stuck with different IDEs doing different things. Is the progress?
It also means that anyone developing a future IDE can no longer support these languages without support. Is this progress?
God forbid someone tries python on a new architecture where new the entire complex toolchain is not going to be available. They just can't use this language. Is this progress?
I don't know how this is a sign of python lacking. Works just fine in jupyter notebooks, too, and that is hardly an "IDE" (though it does have Jedi autocomplete).
If a language cannot be written or used effectively by devs without IDE, it is an incomplete language.
This has happened in the past is Java. Within a few years, Python will bloat up like Java and everyone will move on to greener pastures.
Even with typing, how often do you see Any in python vs Object in Java?
I love statically typed languages. I also love python. But pythons "type annotation" system is garbage. It is neither here, nor there and developers spend an inordinate amount of time figuring out types.
While other languages typing is a productivity booster, pythons type annotation absolutely isn't, especially with complex types.
What has kept you on tabnine?
I'm in "python is not suitable for production" camp, would be happy to change my mind.
> I remember the dark ages, before type hints. It was rough. I lived in the REPL. Write 3 lines, run, Exception. Write 3 more lines, run, NoneType error.
That's why I switched (back) to typed compiled languages and never went back.
disallow_any_unimported = true
disallow_any_expr = true
disallow_any_decorated = false # true if you never plan to use Any
disallow_any_explicit = false # true if you never plan to use Any
disallow_any_generics = false # true if you never plan to use Any
disallow_subclassing_any = true
disallow_untyped_calls = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
disallow_untyped_decorators = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_ignores = true
no_warn_no_return = true
warn_return_any = true
warn_unreachable = true
strict_equality = true
strict = trueThe editor experience only increases that factor as you can navigate a lot more freely. Even for small code bases type hints are a revelation. Knowing if something is nullable or not, alone, has caught 100's of bugs in waiting.
Most of the complaints here feel like attacking a strawman, particularly the ones arguing what's the point if you're not running mypy. Run mypy! It's like saying what's the point of writing unit tests if you don't run the test suite. For any sufficiently large code base you absolutely should be running mypy with any plugins for your framework of choice added.
For heterogeneous dictionaries use `dict[str, object]` and then you're forced to use the existing duck-typing mechanisms you're used to. Or, you refactor into better types and/or structures.
Is there friction? Sometimes. You don't have to annotate everything. If the cost is too prohibitive for a particular structure or function, ignore it. Sometimes there are mypy bugs that make writing correct code impossible. # type: ignore[reason] that sucker. Don't throw the baby out with the bath water.
We use a flake8 linter to enforce that all new or modified functions at least annotate their return type which makes interacting with functions quite a bit nicer, and encourages all devs to at least do the minimum. Usually you find that most args are appropriately typed at the same time.
Its fine though - I'm glad the culture is trending towards understanding that strong typing saves you time rather than causes you extra time. This is true even for very small and simple programs, and even if you're the only developer.
Python type hints are slightly different because there are type hints as a syntax feature and then there is type checking, which can be static like mypy or during runtime like Beartype. Every type checker comes with its own type system, though they are hopefully somewhat similar...
I would even go so far as to say that Java's type system is the very one that left such a bad taste in people's mouth that many people swore off explicitly typed languages for a decade or two. It really was that bad, especially before the last few years. It added a lot of extra boilerplate for minimal practical benefit. People used Java because it was fast, cross-platform, and less likely to go horribly wrong than unmanaged languages like C and C++. I doubt very many people used it because they just loved typing tons of boilerplate or because they thought the type system was good enough to catch tons of errors.
The type systems in Rust, TypeScript, and other more recent languages are far more expressive (and capable of catching real-world bugs) than what Java offered. If you think Java has a good type system... things have come a long way since then. Java gets updated continually, and some people will surely jump at my comment and claim things are great these days, but, no. Java is not up to par, even today. C# has done a better job of keeping up, in my opinion, but even it still lacks extremely useful features like proper Sum Types.
I don't have much experience with Python's type hinting, so I can't comment as directly there.
When a language like typescript supports too many obscure features, or multiple ways to do the same thing, the curve becomes too steep to be proficient.
Does type pattern matching comes close? https://dotnetfiddle.net/Oz2Qyd
Yeah, it doesn't prevent passing unexpected type as object to Print func and getting runtime exceptions. And returning "object" type isn't helpful if we want to prevent runtime errors. But then again, it can be written to spit out compile time errors: https://dotnetfiddle.net/XfKVaa
Do you mean "statically typed"?
To be honest, I think a lot of Python [or arbitrary dynamic language] developers can be quite sloppy, throwing dictionaries around everywhere because it's easy and leading to some really hard to read code. Type hints guide people away from that sloppiness while still allowing access to most of the features that make dynamic languages so useful and expressive.
It's a best-of-both-worlds situation and I'm here for it.
git diff -U0 --relative origin/master... | flake8 --diff --select <custom-codes>
I'll leave the flake8 plugin as an exercise to the reader as ours is intertwined with code I can't share right now.
Though honestly I think it’s worth it to just bite the bullet and spend a full day or two to go through and fix every single error. After this you will have a much easier time navigating code base. I mean, your “new code” is probably modifications to your old code or calling your old code right? So you want to have at least the boundary between new and old typed. It’s quite satisfying anyway, as you will likely learn and discover tons of bugs on the way.
Your code must be quite small.
Python by design encourages duck typing, which means that you'll have plenty of places that simply take multiple different types, and just slapping a union is usually non-trival and not very beneficial either (eg. you'd be randomly patching things with fakes and mocks in tests).
If you think it's a single day job, I am sure my company (and plenty others) would pay you gladly your single daily rate to get our codebase migrated in a day :-D
File by file.
# mypy: disallow-untyped-defs
I did occasionally get pushback on it but stricter checks will be inconvenience at first and I argued it’d help over time. Now that we’ve had it a year it’s pretty well accepted in same way many would accept run formatter.
(git main) > git checkout -b abc-1234/type-fixes
<do work on types and commit it>
(git abc-1234/type-fixes) > git checkout -b abc-1234/feature-branch
<do work on feature and commit it>
A few concrete criticisms:
1. If you're using a dynamic language, then _by definition_ the language will not enforce your static hints at runtime. However, good news! Python has always been strongly typed, and _does_ enforce types at runtime!
2. A number of the examples given would be _impossible_ to statically type in most compiled languages (those without dependent types). It's hard to know where to go with a critique saying that you can't fully represent heterogeneous values in a dict, given that you can't do this _at all_ in many statically compiled languages.
It seems to fall back on saying "use of the type system requires too much discipline to be useful". This might be an interesting criticism on its own; however many current users of Python can say, from experience, that the required discipline is not a high enough hurdle that it prevents us from using types consistently and correctly.
There is plenty to critique about Python's static typing, but this article would have been better titled "I'm confused about Python's static type system and want somebody to show me how it's actually used in real-world scenarios".
Duck typing and heterogeneous dictionaries are (and have been) standard Python.
Adding a type system which doesn’t respect duck typing by trying to access the member even when the types don’t match or which can’t express standard idioms used in Python seems a poor architectural choice.
It’s not saying that Python’s type system is “too much discipline”, but rather that the type system doesn’t encode typical Pythonisms.
and heterogeneously typed ducts: https://peps.python.org/pep-0589/
Traditionally, duck typing is used to inject types into a library that isn’t expecting extension at that particular point — eg, substituting a test class for a real class in a data object that normally wouldn’t be a protocol.
I’m not seeing how that PEP addresses that use case.
- - - -
Similarly, the heading on the dictionary says “for a fixed set of keys” — but what if I want a dynamic heterodox dict? Eg, unpacking JSON.
You just end up shoving “any” all over the place. At which point, are the types helping?
Though, protocols do help the dict case for typing:
dict : Hashable -> Any
https://peps.python.org/pep-0544/#:~:text=Structural%20subty....
> substituted-for class to be defined as a protocol.
... which is false
> Similarly, the heading on the dictionary says “for a fixed set of keys” — but what if I want a dynamic heterodox dict? Eg, unpacking JSON.
You can quite obviously decode to recursive types (e.g. using pydantic), not sure what the problem is.
This section seems to agree with me, where it explains why normal classes can’t be subclassed to protocols:
> Now, C is a subtype of Proto, and Proto is a subtype of Base. But C cannot be a subtype of Base (since the latter is not a protocol). This situation would be really weird. In addition, there is an ambiguity about whether attributes of Base should become protocol members of Proto.
https://peps.python.org/pep-0544/#protocols-subclassing-norm...
- - - -
I’m not sure why you think it’s “obvious” that you can use a third party library to solve the problem — or why that addresses my complaint that the built-in type system doesn’t work for that.
If anything, the existence of a third party library hints the standard library doesn’t cover the use case.
edit: To be more precise: The article specifically mentions that accessing invalid members give you a type error, but posits that people will probably not bother and just use 'any' instead. How is that not saying 'too much discipline'?
I’m also generally pro-types, but I think it’s worth having a discussion about this type system in the context of Pythonisms.
- - - - -
If you’re using “any” for most of your types, then it’s not providing value — an untyped statement implicitly has the type “any”.
There’s a reason I brought up the JSON example: loading and manipulating JSON of varying structure is something I do a lot at work.
>There’s a reason I brought up the JSON example: loading and manipulating JSON of varying structure is something I do a lot at work.
Do you have a specific example that you find troublesome? For JSON with a fixed, regular structure (i.e. no subtyping) something like Pydantic works well, otherwise using unions + protocols with custom validation code has covered even my most esoteric use cases so far.
If we're talking about unstable JSON, a recursive union type works well enough in my experience, though defining that type gets a bit ugly if your type checker doesn't directly support recursive types.
Python types don’t support a common Pythonism used frequently in my work.
In fact, you admit that even fixed JSON can be difficult without a 3rd party library.
You’re lecturing me like I don’t understand types without realizing that you repeated my point about a gap in the current type system.
JSON = Union[
None,
bool,
int,
float,
str,
List['JSON'],
Dict[str, 'JSON'],
]
>In fact, you admit that even fixed JSON can be difficult without a 3rd party library.
Sure, but that's just because the standard library (for now?) doesn't offer deserialization with runtime validation. Java doesn't have that either, if you want to parse JSON you'll have to either roll your own or rely on a third party library.
Don't get me wrong, the situation isn't perfect, but especially if you use the 'right' tooling, you can in my experience already get a perfectly serviceable experience, which to me is indicative that the type system itself is fine and pythonic, it's just the tooling that is lagging a bit behind at the moment.
> Sure, but that's just because the standard library (for now?) doesn't offer deserialization with runtime validation.
> it's just the tooling that is lagging a bit behind at the moment
So… the standard library has exactly the problem I articulated, and you’re violently agreeing while speaking down to me.
I think that reflects poorly on you.
If you think that's violently agreeing and talking down to you, then there must be some communication barrier that I don't know how to overcome.
This does not define the type of a valid json object. (fe None is not a valid json object, neither is '')
However, even in Typescript it is nontrivial to _annotate_ these complex types, and in most cases it's still possible to do in Python - you just need to commit to using something like dataclasses rather than pretending that what you have is a `dict`.
Basically, to get the most out of static typing in the world of Python, you do have to write more boilerplate than you would in Typescript, and in particular you need to avoid dicts and prefer various sorts of class-based data containers (again, dataclasses, attrs, namedtuples, etc).
The only one which stands out to me as non-trivial in TypeScript is thrown exceptions… and even then only because you can only type them in a catch clause, and only with unknown, and they’re invisible to callers. But then this is why people who would care for checked exceptions quite reasonably just use something like a Result type.
But I agree with your conclusion completely, and it’s basically the same one I came to reading the post.
I was curious about how the current state of things compares to TS, as I may soon have a foray back into Python soon, and would be delighted to bring some static types into the mix. Pretty much as I expected, it’ll probably be more effort/less valuable than TS, but more valuable to me than to the author.
— Consider using Pydantic (either its drop-in dataclasses replacement, or its models). It has issues and documentation is lacking in places, but in many use cases it’s a boon when it comes to being confident about data you read and/or pass around.
— Don’t use TypedDict if you foresee needing anything like optional keys, especially with defaults. I went with TypedDict and it came back to bite me later (though was relatively easy to refactor).
— Keep an eye on https://pypi.org/project/typing-extensions/, it has some useful utility types. I wish I knew about this package a couple of months back.
— If you use VS Code (which I started after switching to TS), there’re some shenanigans to be aware of. Unlike TypeScript’s elegant approach where you’re in control of installing typing versions, VS Code’s official Python extension will force-install unvetted third-party typings, not caring if they don’t match library versions in your use and not allowing to opt out. In later versions you can unset mypy in Python extension settings, and use a separate Mypy extension that doesn’t do this behind your back and doesn’t send you deep into some rando’s typings when you hit F12 to jump to definition.
— That aside, VS Code can provide development experience is not that far from TypeScript. You can keep a virtualenv nearby and select its Python bin as interpreter in your workspace, giving Mypy access to dependencies (and if VS Code’s interpreter selection GUI resolves symlinks, you can enter interpreter path by hand so that it doesn‘t).
— Sphinx documentation is good at picking up typing hints and works reasonably well with Pydantic’s models, too.
This seems like an inaccurate and condescending put-down. What is the definition of "educated" in this context? Is there a book or generally well-known resource?
The author is clearly thoughtful and curious. Near the top of the post he says "what I’m hoping for is that someone will come along and tell me that I got it all wrong. “When you do it as follows, the system works: $explanation” Because, you know, I’d really like to have static typing in Python."
> 1. If you're using a dynamic language, then _by definition_ the language will not enforce your static hints at runtime. However, good news! Python has always been strongly typed, and _does_ enforce types at runtime!
And yet, the author points out that this is a working program:
foo: int = 'hello'
print(foo)
Can a string be printed? Yes! Then Python is (correctly) allowing typesafe behavior here. Your (incorrect) annotation does not in any way contradict the typesafe-ness of printing a string (or an int). They're both equally printable.
This is just a very, very bad example, plain and simple. It 'looks' bad to a superficial reading, but in practice it demonstrates only what is already known about static typing in Python - it is enforced (or not) separately from the interpreter.
This said, I find the linting/type-hint-checking stage of Python cumbersome and confusing :(
Am I the only one baffled by this way of thinking?
Static type annotations by definition are not enforced at runtime. This has been true of every language that has ever used static typing, including C, C++, Java, Rust, Typescript... you name it, statically typed languages only enforce their static typing at compile/analysis time.
Yes, it is possible to annotate types in Python incorrectly. It's possible to do this in all other languages that allow type-unsafe behavior (whether natively or via reflection, etc). This may be less common in some languages than in others, but it is fundamentally possible in the vast majority of languages that perform static typing, because those types are fundamentally enforced at analysis time, not runtime.
The author of the article seems to be unfamiliar with these distinctions, and maybe you are too. It's fair to complain that these distinctions are confusing and make things harder for programmers. Nevertheless, very few languages have ever been designed with a type system that acts exactly the same at analysis time and runtime. It's a very difficult problem, partly because these are all abstractions from the perspective of the underlying computer, which only understands boolean logic and integer/floating point math, and has virtually no other notion of types in any formal sense.
Type systems are a complex topic, and as someone who has been paying attention to them for a while, it's frustrating to see uninformed discussion of them show up on Hacker News. That said, it's perfectly understandable that people are confused by these distinctions, and rest assured there's a lot of effort going into developing better languages that suffer _less_ from these issues.
For the everyday working programmer, however, there are currently lots of tradeoffs to be made, and Python's approach to static+dynamic typing is actually pretty usable, all things considered, which is why the community overall has embraced the new static typing despite its blemishes.
It is pretty damn hard to make this mistake in languages that enforce static typing. I mean, you would have to go out of your way to do so. In Python, however, it is trivial to write the wrong type signature or to modify the body of a function so that it no longer matches the signature.
"Fundamentally possible but terribly unlikely, as opposed to the Python way of doing it" would be a more accurate description.
Since this argument is not meaningfully different from "comments that are lies considered harmful", it seems fair to expect reasonable people to dismiss it as uninteresting.
Do you think that's a fair treatment of someone who disagrees with you but has been respectful of your opinion so far?
I realize proglang debates are flamewar territory. But I have been very careful to state I find Python type hints puzzling and less useful than they should be. It is obviously an interesting opinion shared by many others, not just me or the article's author.
As a fan of statically typed languages, I do indeed find some Python programmers engage in a form of Stockholm's syndrome. It usually takes the form of the assertion "I never found a bug that was a type error". Have you or anyone you know ever said this?
In my mind, the opposite statement of the one you attribute to me would be "wishful thinking and a positive attitude is enough to catch mistakes, it's not necessary to have help from automated tooling". In this day and age, I cannot disagree more.
It's precisely because I do like and use them that I find Python's static type hints to be extremely useful for large Python codebases.
What I hear you saying is "Python does not offer these things". What I hope you hear me saying is "Python does, in fact, offer these things".
> I think your (and the article's) argument could be summarized as "static type annotations without automated enforcement by actually running a type checker considered harmful"
It illustrates your mindset, in my opinion. I think type annotations without checking them (or without a satisfying implementation of said type checking) are mostly pointless.
You also left out the part where I remarked on the dismissive tone of your reply.
However, there are other aspects to the article’s argument, such as the fact that it’s easy to end up with type annotations going silently unenforced even if you do run the checker.
Also, some languages like Java do actually enforce types at runtime. If you've ever called a Java method dynamically with the wrong types, you'd know. You can run into ClassCastException, or worse (ClassNotFoundException if you have the wrong classloader.) This is a runtime error.
A type system is just a set of rules that must be followed. The rules can be as weird or as simple as you want. Just because <int> + <string> is allowed doesn't mean it's not enforcing types.
Not really a criticism, just trying to push to expand your idea of what a type system is.
Is it "legally" baffling, i.e., do I understand why this behaves in the way it does purely mechanically, and do I understand the arguments the gymnasts are making? Yes.
But from an idiomatic, colloquial, linguistic, syntactic, programming-historical, or programming-cultural standpoint? Baffled.
Even calling it gradual typing is baffling. It's just adding metadata-that-doesn't-look-like-metadata that sophisticated programs that aren't part of the Python bytecode compiler can use. The definition I know is the same one from Wikipedia:
> Gradual typing is a type system in which some variables and expressions may be given types and the correctness of the typing is checked at compile time [...]
This is definitely not what Python is doing.
Maybe not by default but there is tooling that makes it work that way — that’s exactly how the python build system works at my company. I don’t know the details but when I “build” python code (creating bytecode pyc files) it produces compiler errors and fails to build if I e.g. try to pass an int to an f(x: str). This has usefully caught bugs before I ran some expensive/time-consuming scripts.
I guess you could say this doesn’t count as it’s a separate program doing the type checking but IMO there’s not a clear distinction between that and e.g. having a first type checking pass in the “same” compiler program.
> While these [type] annotations are available at runtime through the usual __annotations__ attribute, no type checking happens at runtime. Instead, the proposal assumes the existence of a separate off-line type checker which users can run over their source code voluntarily.
When I read "compile time", I mentally replace it with "before runtime". There isn't a real compilation step in Python, so it's reasonable to accept static code analysis as part of "compile time" in that definition.
The point is that type annotations will let a static analysis tool make sure that the program is correctly typed. This can be enabled easily in most of the modern IDEs and editors. You could just decide to block the code's deployment in CI/CD if the type checker raises issues, for example.
E.g. this kind of code: https://play.rust-lang.org/?version=stable&mode=debug&editio...
typically works fine (or worse, works fine in debug mode and causes unpredictable UB-related bugs in release mode).
std::mem::transmute here is a no-op at runtime; all it does is subvert the static type checker.
The difference between statically-typed languages and things like Python with type hints is _not_ that the former has any runtime type checking; it’s that subverting the (static!) type system is more difficult and requires more obvious ceremony, and that static type checking is required whereas with Python it’s optional.
If you hacked rustc to disable static typechecking you would get a similar experience to Python with type hints but without any mypy-like tools. (Of course, this would be hard to do in practice, because Rust, like most statically-typed languages, uses types not just for typechecking but also for code generation and method dispatch.)
When Python "enforces types at runtime" this is the actual types, not type hints. Type hints are not a runtime artifact. The "actual type" here is "string", and enforcing it means Python would not allow invalid operations on it without error'ing. A programming language that will let you do basically anything to any value because it doesn't enforce type safety is of course C. Python is safer than C.
*(unsigned long*)0xFFFFFF14 = 0x749235f8;
*0xFFFFFF14 = 0x749235f8;
char* s = "abc";
*(unsigned long*)0xFFFFFF14 = s;
Best of all, all of these are enforced at compile time.
Now, C absolutely will let you convert an int to a pointer, or a char to an int, or an array to a pointer, or... well, it will let you convert lots of things to lots of things. Some of those things are unsafe unless you are quite sure what you're doing.
> A programming language that will let you do basically anything to any value because it doesn't enforce type safety is of course C.
False on several grounds. You can't call something that isn't a function. You can't call a function on something that doesn't have it. You can't use an int as a pointer or an array without casting it. You can't use an int as a dict (not that C has any built-in idea of what a dict is...)
> Python is safer than C.
Depends on what you're doing, and what kinds of errors you are more prone to.
You know what I meant and what error I was clarifying for the comment I was replying to. Yet you went out of your way to point all sorts of irrelevant mistakes in my post, when the gist of it was right.
Do you feel it was more important to correct me on these trivial things, or was my reply more or less correct when fixing the conceptual error in this sentence?:
> "In what reasonable sense can Python be said to "enforce types at runtime" here?"
Here's a nitpick of my own to your post:
> "Some of those things are unsafe unless you are quite sure what you're doing."
Wrong. Type (un)safety does not depend on you "being quite sure of what you're doing".
PS: assume I know C and that I've programmed complex things using it. No need to catch yourself with statements like "not that C has any idea of what a dict is". Assume we are both C programmers. It will make you sound more polite.
2. Your last two sentences were, I think, not needed for the point you were making. They were also somewhere between gross generalizations and flat-out wrong, and felt like a gratuitous slam on a language that wasn't even the topic. I thought that deserved a response, even if it wasn't your main point.
3. I may assume that you know C. I don't assume that everyone reading this exchange knows C.
Pointless correction. Gross generalizations are not wrong in this particular situation, which is not meant to teach anyone C.
Let me try again.
Do you think that your correction was more important than helping the poster of this comment correct their conceptual mistake?
> "In what reasonable sense can Python be said to "enforce types at runtime" here?"
Sure you can. That's exactly what you do when you call a function returned by `dlopen` -- it might be a function, but it might also just be garbage. It might even be garbage that's coincidentally marked as executable and does some stuff before eventually crashing!
> You can't call a function on something that doesn't have it.
This is also very easy to do -- you can default-initialize a structure containing function pointers, and then call one of them. You'll be calling some random crap on the stack, which might or might not do anything of interest. But either way, C will happily let you do it.
C isn't completely untyped (there are, as you've observed, things it will forbid at compile time), but it's about as weak as a static typing system can be.
In both examples, you have tell the compiler exactly what the types are. In the first one you have to cast the return value of `dlsym` to a function pointer, and at that point, as far as the compiler is concerned, it's a function. In the second example you have to explicitly declare the struct containing function pointers, so of course you can call its members.
So it's strange to suggest that C is somewhat untyped; it's not untyped at all. C just makes it very easy to force the compiler to assume different type for a value (that's necessary for low-level code like the examples in the comment you replied to; and of course that makes it very easy to do unsafe things, I don't think anyone disputes that).
And about the "weak typing system" comment, note that nobody agrees on what exactly that means. Just as an example, see how this page[1] defines "weaker" vs "stronger" types, and how it's completely different from the way you're using here. In general, I find people call type systems that don't do what they expect or want "weak", but that's not an useful discriminator: at that point you might just call it "bad typing" to dispel any illusion that it has any objective meaning.
[1] http://book.realworldhaskell.org/read/types-and-functions.ht...
Just because "weak typing" means something different in Haskell doesn't mean it can't be used meaningfully (and beyond "bad") in the context of C. C's typing is historically referred to as "weak" because C's notion of casting doesn't distinguish between type and value transmutation: pointer-to-pointer casts don't convert the referent (because they can't), which in turn gives the C compiler very little leeway in proving that the program's types as declared have any particular meaning at runtime.
Compare this to Python, which is "strongly" typed in the same sense: doing `y = str(x)` on `x` means that `type(y)` actually is `str`, and not merely a promise to the runtime. It's enforced, which makes it strong.
That's why it's useless to say things like "it's about as weak as a static typing system can be" and "has been historically considered weak".
If you had explained your objection to C's type system (like you have now), I wouldn't have said anything. And I've been on the Internet long enough to have seen C's typing is historically referred to as "weak" because X for *many* values of X, so I disagree that that's the only or even main objection.
I believe this is the relavent passage in the article the GP is referencing, which is incorrect if you take "runtime" to mean something like when the line is executed: "Python is a dynamically typed language. By definition, you don’t know the real types of variables at runtime."
It _is_ however correct to say you don't know the real type _before_ runtime, at least python does not.
The Any type is to allow incremental typing. You start out with dynamic code, and progressively 'typify' it by adding more and more annotations, with Any working as a stopgap where for the moment no valid type exists, until you finally have a fully typed program. Inserting Anys because you're lazy is like casting things to Object in Java because you're lazy, abusing it like that is just incredibly sloppy and bad code.
If someone is really struggling with this, simply use a linting rule to ban Any in 'mature' code.
There are lint rules to avoid typing as `any`.
Python is both strongly typed and protocol-typed. In the case of `print(...)`, the protocol is that the received parameter responds to `str(...)` (i.e., has `__str__`).
(What Python isn't is statically typed. But "strong" and "static" are completely different typing dimensions.)
> In what reasonable sense can Python be said to "enforce statically declared types at runtime" here?
which is certainly an extremely pertinent question for a language that allows declaring supposedly static qualities of a program.
It can't, because Python doesn't have statically declared types. It's a strong, dynamically typed language.
Python also doesn't allow you to declare static types. It allows you to declare type hints, which are a sort of adjacent and potentially unrealistic (or just plain incorrect) typechecking proof that doesn't correspond at all to runtime behavior.
Python has the capacity to have some form of gradual typing built-in at the language level, but it does not, which has led to the enormous confusion we have before us. The SBCL implementation of Common Lisp is a great example of a strong, dynamically typed language that is able to, when possible, detect errors statically at compile time. That tech is 30 years old now.
There is no reason why the Python bytecode compiler could not statically or dynamically detect this programmer error at either compile-time or run-time present in this statement:
x: int = "x"
(declaim (type integer x))
(defvar x "oops")
; ==>
Unhandled SIMPLE-TYPE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {10005D05B3}>:
Cannot set SYMBOL-VALUE of X to "oops", not of type INTEGER.
quitting
compilation unit aborted
caught 1 fatal ERROR condition
I don't see why that would be the case. The value of Mypy is that it's essentially a proof language over something that resembles Python[1]: it's a way to do some amount of static typing without having to maintain a separate copy and representation of the program.
And sure: Python could detect inconsistent type hints at runtime. But why would it? It's already strongly typed, and the presence of type hints usually means that someone is already running Mypy or another checker during development. It's not clear that there's a significant advantage to be gained, particularly one that justifies the additional overhead.
[1]: "Resembles Python" because mypy does not actually evaluate any Python. It doesn't know what types your program has at runtime; it only knows type hints and a few small rules (for things like string literals) and trusts the developer to reconcile those rules with Python's runtime behavior.
Because the programmer explicitly intended a contract which is being violated, which would in turn indicate a bug.
Anecdotally, my code in python is functionally fully typechecked and runtime type checking would solely penalize me by slowing down my code. There's no need for it because the type checker correctly proves the soundness of the program's types, in exactly the same way that Java's or C++'s do.
The point is precisely that you don't need to suffer the costs of runtime type checking if you have a solid typechecker. Lisp's approach here is worse than the one adopted by python/js/ts/go/etc.
Yes, but is it caught at static analyzer run? Yes, I typed it into vi and ran mypy on it and it said "error: Incompatible types in assignment (expression has type "str", variable has type "int")."
But I didn't have to run mypy on it, my editor told me: "Expression of type "Literal['hello']" cannot be assigned to declared type "int"".
Maybe the word "educated" wasn't the right choice in this case, can you offer a better one? I honestly couldn't make it through the entire article, despite several attempts. It is full of inconsistencies and what seem like wilful misunderstandings to justify the authors conclusion.
Particularly as the author starts off saying "run a static analysis tool" as if knowing that's how you go about it, and then saying "it doesn't catch it at runtime".
It does: in form of TypeError exceptions. It doesn't enforce type hints, sure, but that's why they're called hints and not declarations.
In [1]: foo: int = 'hello'
In [2]: foo + 1
TypeError
Input In [2], in <cell line: 1>()
----> 1 foo + 1
TypeError: can only concatenate str (not "int") to str >>> foo: int = 'hello'
>>> type(foo)
<class 'str'>
But we have (to quote another commenter) "uneducated" people making a supposedly boneheaded mistake to assume
x: T = y
Instead, this line is more properly interpreted as "documentation reasonably believed to represent a type saying something about values that x can be assigned at runtime, that's not in a docstring, that some programs may query."
I can totally forgive anybody who believes the former over the latter, and expresses their belief in a statement like that which you've quoted.
Absent of active machine checking (e.g., via mypy), type hints should be considered exactly what you said: another form of documentation, one that might just be wrong.
But in Python, it's not (and documented in a PEP as not), and it causes massive confusion.
[1] https://devblogs.microsoft.com/typescript/a-proposal-for-typ...
Python is strongly typed. It respects types at runtime e.g., `"1"+1` causes TypeError. Python is not statically typed.
For comparison, C is statically typed but it is weakly typed e.g., printf function has no idea about actual types of its arguments (in particular, errors in the format arg may produce non-sensical results silently).
As far as I remember, the "__str__" method is present on all objects in some form, so print will happily take any type.
Most people I know writing Python don't use type hints because they are too much of a hurdle for very little payoff. The tooling that pays attention to type hints is slow as molasses or difficult to use or understand. The type hints themselves are of dubious use.
As a fan and advocate of static typing, I find myself advocating for type hints anyway, but I must agree I often can't reply anything to my coworkers' objections, because Python type hints are truly not that useful.
I use mypy daily on big codebases, it is fine, not fast, but fine.
> The type hints themselves are of dubious use.
They tell you when you make type errors, they help you understand what type things should be. The same as in literally every other language with static typing. There is nothing special here, nothing different. python with a static type checker running in strict mode is not fundementally different from Java's static type checking.
> As a fan and advocate of static typing, I find myself advocating for type hints anyway, but I must agree I often can't reply anything to my coworkers' objections, because Python type hints are truly not that useful.
What do they lack that would make them useful?
No, obviously not the same, otherwise I wouldn't be complaining. They are not even on par with Typescript, which I'm not a fan of either.
> [type hints] tell you when you make type errors
Not according to other comments I seem to be getting here. Other people are arguing type hints are not primarily for checking, but a form of notation for documentation. Seems wasteful, and I wish the Python community and tooling decided instead that they are for actually checking them.
> What do they lack that would make them useful?
Standardized, go-to tools that work in the build pipeline and that catch most errors without taking a long time to do so.
I haven't found mypy to fill these requirements. It's so bad I cannot convince my coworkers to make the effort to write more type hints.
What is the difference?
> They are not even on par with Typescript, which I'm not a fan of either.
Go is not on par with Typescript or Python, I still don't think it is okay for people to just say fuckit and `interface{}` it all and it is still shit to work with code that does use `interface{}`. At least Python with mypy has null safety, something that Java and Go does not have. There are some places it is worse than other statically typed languages, others where it is better.
> Standardized, go-to tools that work in the build pipeline and that catch most errors without taking a long time to do so.
It is mypy. What actual type errors does mypy not catch that you want it to catch? Why can't you use it in the build pipeline? I do it every day, it catches all the errors it should. The one complaint I can maybe see is the duct type compatibility complaint, and it is not really something that comes up that often for me, and definitely not something I would say invalidates the whole concept.
I'm not going to repeat myself, I already told you.
I find mypy slow, unsatisfying, inconsistent, and it fails to catch many type errors. No, I'm not going to go look in my work laptop to give you an example.
> There are some places it is worse than other statically typed languages, others where it is better.
In most places it is way worse, and I'll find it very hard to find common ground with anyone who disagrees on this.
Feel free to disagree, but I don't find this conversation useful.
Compare that to a c++ project where it’s not unusual to have 10, 15, 30 min build times.
Or even worse, the time to wait for discovering the problem runtime in prod.
The type hints also help in IDE navigation, where there is zero time to wait, pycharm can go to definition or find references in under a second.
That's locally on a fast (ish) laptop. GH CI takes 10 minutes to run tests, so I'm a little hesitant to see how long mypy takes to run.
in my experience it's a far better type checker than mypy, which tends to silently not check things without you ever realising
> Python is a dynamically typed language. By definition, you don’t know the real types of variables at runtime. This is a feature.
In "you don’t know the real types of variables at runtime", this is just flat wrong. One doesn't know the real type of a variable until runtime. It seems to me the author maybe has a bit of confusion between weak typing and dynamic typing here.
I read the sentence as
"you don't know the real types of variable at [the start of] runtime"
"you don't know the real types of variable *during* runtimes"
In the program
def x(f):
return f(3)
If your answer is that `f` has type "function-like", then that's a much weaker kind of a type than even type hints offer, let alone a real type system.
We can tell by looking at what python triggers TypeErrors for.
>If your answer is that `f` has type "function-like", then that's a much weaker kind of a type than even type hints offer, let alone a real type system.
f implements the Callable interface - meaning it's either a function/method or an instance of a class with a `__call__` method.
So you can pass e.g. `print` or an object with a `__call__` method. If you try to do `x(5)` that's an immediate runtime TypeError because `5` can't be called.
f must also accept one int argument. It can have additional optional arguments, and it can also accept other types (e.g. `f("foo")` can also work!), but it must work on one int. The one int can already be optional - the function can also work with zero arguments.
Calling e.g. `"foo".join(5)` fails with a TypeError because 5 isn't Iterable, and calling `"foo".join([1], [2])` fails with a TypeError because it gets more arguments than expected.
Counterexample: PHP is a dynamic language which enforces static type declarations at runtime.
The point is that it is not expected by definition, since by definition, static != runtime.
So PHP is not a dynamically typed programming language according to you? It enforces type hints at runtime just fine.
There is nothing in the definition of dynamically typed languages that says they can't use type hints to perform runtime checks.
What statically compiled language doesn't provide a way (sometimes clunky) to have heterogeneous values in a dict? You can do that with void* in C, even
What definition is that? Raku is a dynamically language with gradual typing which will enforce those types at runtime (if it cannot do so at compile time).
I don't think this follows by definition. It's common not to check the static hints at runtime, but a dynamic language could choose to treat them as contracts or assertions. Over in Lisp-land, SBCL will enforce static type declarations at runtime [1], contrary to most other Lisp compilers, at least under the default compilation settings (you can force it to skip the checks by compiling with a low "safety" level). If the compiler can prove that a static type declaration always holds, it will omit the runtime check; otherwise it will compile it into a runtime assertion.
This isn't really true. For example, in Java it's Map<K,Object>. For languages without subtyping, existential types are often used. It's quite common to see in infrastructural code, e.g. caching.
1. There is no way to get typing like dataclasses without writing your own mypy plugin. The syntax is simply not expressive enough to do it. This means that if you want to be productive with a library like Pydantic, you also have to add the mypy plugin to your dependencies and add it to your mypy config. Otherwise, no typing.
2. You cannot compose types. There is no way to say "hey this type is the same as this dictionary here, except the keys are all optional." If you're trying to type a RESTful API, your only option is to repeat yourself and carefully keep all your types in sync.
3. To this day, there is still no way to express optional keys. Not "Optional" keys, but keys that can be left out of your dictionary. The closest you can get is this weird hack where you can set `total=False` in your TypedDict, which makes all keys optional.
I really wish Python learned from Typescript, but it's too late at this point. Too many half-baked measures have already made it into the spec.
This would be solved by PEP 681, which originates from (and is already used in) pyright: https://peps.python.org/pep-0681/
> 3. To this day, there is still no way to express optional keys. Not "Optional" keys, but keys that can be left out of your dictionary. The closest you can get is this weird hack where you can set `total=False` in your TypedDict, which makes all keys optional.
This is solved by PEP 655, which introduces NotRequired: https://peps.python.org/pep-0655/
It's implemented in mypy 0.930: https://mypy-lang.blogspot.com/2021/12/mypy-0930-released.ht...
> I really wish Python learned from Typescript, but it's too late at this point
Personally, I feel like it's partly just entirely different styles: Python relies on a mix of nominal and duck types, whereas JavaScript is much more focused on structural typing. As a result, there are a lot of things like TypedDict that are absolutely essential for JS to even function but a decent bit less so in idiomatic Python code (keeping a significant amount of dicts around uses much more memory than objects, I recall a PyCon talk explicitly mentioning this too).
It's not exactly elegant (hence PEP655), but it works just fine.
1. It reduces the overhead to use your interface. Your users don't have to import your struct classes, and can just pass in bare dictionaries. If Python's typing was more effective, these dictionaries would also be type checked to make sure you're calling the library's APIs correctly.
2. It deliberately closes off the option of adding state or initialization logic. Classes let you do way more than just encode a struct, and as a codebase grows they have a tendency to become more complex just because they can (like a gas filling its container). This makes APIs harder to maintain and work with. Sometimes, all you need is "raw structs".
import typing
class Foo(typing.TypedDict):
a: int
def do_thing(value: Foo) -> None:
return
foo = {"a": 1}
# error: Argument 1 to "do_thing" has incompatible type "Dict[str, int]"; expected "Foo"
do_thing(foo)
interface Foo {
a: number
}
function doThing(value: Foo): void {
return
}
const foo = { a: 1 }
doThing(foo)In the Python case, "foo" and "Foo" might look structurally compatible, but they aren't. "Foo" isn't just a type, it's an object of type "class". You can for example print(Foo), you can't console.log(Foo). "foo" is not an instance of the Foo class. What you are looking for is "dict[str, object]".
import typing
class Foo(typing.TypedDict):
a: int
def do_thing(value: Foo) -> None:
return
foo: Foo = {"a": 1}
do_thing(foo)
Typescript's type system, of course, had and has the advantage of being a clean sheet design with an intermediate compilation step, so the comparison is hardly apples to apples. But there's no denying that I wish frequently that Python could have opted for a more powerful static type system.
Like the author, I've mostly given up on adding type hints in my Python code. I now only use them when I want to help my IDE find autocomplete suggestions.
Whereas TypeScript was a game changer for JavaScript. I used to hate JavaScript, but somehow TypeScript has become one of my favourite languages! How has the advent of Typing has changed my opinion on these two very similar languages?
- JavaScript without types is a mess, whereas Python comparatively was much better, esp since it does runtime duck type checks.
- Python type hints are much similar to Flow type hints in JS, which I tried, but ditched for the same reasons as Python type hints.
- I was hesitant to try TS's all in approach, cause it was harder to introduce into a project, but after having converted a number of projects to TS, I can see that going all-in is a much better approach than just adding hints as you go.
- TS does checks at many more levels. Eg, if a property is optional or could have different types, it is a syntax error if I don't check the value is valid before use.
- TS does an amazing job of auto-detecting types, so most of the time you don't need to specify types, and it enforces these just as if you declared them.
- TS has reached the critical mass were most popular packages now include type definitions, I very rarely have to add @types/* anymore. This means you get full intelisense on all 3rd packages! I spend a lot less time referring to documentation now!
In hindsight, compiling out types is a great work flow. TypeScript is so good that it has made me enjoy Python less. If there was ever a Python equivalent to TS which reached a critical mass of support, I'd jump all-in in a heartbeat.
For example an optional requires a typing.Optional import or a an ugly "| None" instead of a question mark like TS has. And good luck trying to annotate some complex / nested json, you'll need a bazillion intermediary classes.
The following code is valid:
function foo1(value?: number) {}
foo1()
function foo2(value: number | undefined) {}
foo2()
This is already invalid:
function foo1(arg1?: number, arg2: number) {}
function foo1(arg1: number | undefined, arg2: number) {}
foo1(undefined,1);
https://www.typescriptlang.org/play?#code/AQMwrgdgxgLglgewqB...
This conflation of "optional" and "undefined-able" is more obvious in interfaces. This is why TypeScript added the `exactOptionalPropertyTypes` option:
https://www.typescriptlang.org/tsconfig#exactOptionalPropert...
Sometimes "is not defined" needs to be treated differently than "is set to undefined"
Pylint is closer to Flow than Typescript.
Let's emphasis the "gradual" part. The more libraries are carrying type annotations, the more helpful IDEs will be and the easier it is to type-check your own code.
There are a some competing languages for the Python runtime as well: https://github.com/vindarel/languages-that-compile-to-python
My team began using type hints and aggressively applied mypy to new code starting a couple years ago and it _has_ helped immensely. However, we also started doing some work in Typescript around the same time and the difference in developer experience and code quality is pretty clear. Typescript is just a better solution to the problem of adding type information to a dynamic language.
What's especially frustrating is living through the design and early life of asyncio (Guido, you should have adopted the gevent model), then getting excited about typing, only to find it is also not very good. What's _especially_ frustrating is that Typescript existed two years before PEP 484 was published, so the core developers had ample time to seek inspiration elsewhere, and they came up with a sub-standard solution.
Python needs a Typescript analogue, with a compiler that targets vanilla Python.
Every project I interact with that uses type hints runs mypy as part of CI - and I can see that they're setup to do that by looking at the CI configuration (which these days is usually done using GitHub Actions).
I wouldn't add type hints to a project without also configuring mypy for it.
However, type annotations still make the code so much easier to work with, because they serve as documentation and are understood by the IDE. When I hit "." I actually get correct autocompletion because of the type annotations. When I want to know what a function returns I don't need to dig through multiple levels of function calls.
The article seems to be suggesting that because the type annotations might be wrong that they are worthless, which seems like a ridiculous argument. Any documentation can be wrong, but clearly having documentation that is 99% accurate is better than not having it at all.
In a large code base, our team has found type annotations and mypy to be quite helpful. If you are writing a quick-and-dirty script, forget about wasting time with type annotations. However, for long-lived and important code, it seems to be worth adding the types.
Another thing I noticed, if your type annotations are hard to write, e.g. lots "Union", "Optional" or deeply nested type expressions, it's probably a sign that your code is poorly designed. Think about what types you actually want to consume and return. Define new containers using dataclasses or attrs to keep the types simple, if required.
Rather than maintain all of the mypy stubs for no benefit, we decided to just scrap mypy.
If I added MyPy to the CI/CD checks, some coworkers would flood the code with these to not have to think about their typing. Too bad that some libraries like SQLAlchemy make this mandatory sometimes. I'd like the option to ignore these "type:ignore", but the type system is too immature and has to be overridden sometimes.
https://github.com/microsoft/pyright
https://marketplace.visualstudio.com/items?itemName=ms-pytho...
Overall I do think type hints are worth it though, for maybe two benefits. They force you to look at the ridiculous types you are using, like `List[Union[None, str, List[Dict[str, str]]]]` which is the sort of thing that happens in codebases all the time. It adds just enough friction to push people to make explicit dataclasses or simplify function returns, which is good. Secondly they help with tracking functions which return None, which is a pain when following callstacks in big codebases.
That said, I think pytype makes a lot of sense since it infers types from code, which you can edit by hand and then merge back into the python file when your code is stable.
This seems like an extreme overstatement and a sign of a very weird team environment.
Are you not running your linting and validation locally?
Heck, when dealing with some poorly-typed external code, I run mypy manually to help me figure out what the stupid types I need to assign even are. This is far from ideal, but in lieu of a better solution, it works for me.
I just find the idea that people would be adopting typed Python and then completely ignoring the type-related tooling a bit wild.
And sure, libraries can have bugs, you can open yourself up to big holes with Any... and in cases where I'm super worried about things like that, I'm generally not using Python. But I'd rather use Python with type hints + use the tooling in a sane way then not use the type hints at all.
> Even if the Python runtime did check all the type hints at runtime, then it would still be too late. I don’t want a fancy type exception at runtime. That already exists (most of the time). I want to know about type mismatches in advance.
You should check out Typeguard [1], which lets you add a @type_checked decorator to anything and get runtime type checking that's far better than e.g. a random blowup when you split() on an integer. It does introduce a significant amount of overhead, but it's still useful during development alongside type hints to avoid some of the pitfalls from this article.
In another vein, Any should definitely be used sparingly. For the foo() function outlined in the article, Union[str, int] offers more precision. In fact, you could even argue that using Any is a code smell.
Overall, though, I agree with most of the sentiment in this article. I find myself using type hints more as documentation than anything else, since their true ability to prevent type-related runtime bugs is very limited. Documentation has the same qualms the author outlines, anyway:
> You can not be sure that they are correct. [...] They waste mental energy when reading code, they create new maintenance burdens, and they are potentially deceiving: You cannot trust them.
But they're still slightly better than docstrings...
[1]: https://typeguard.readthedocs.io/en/latest/userguide.html
What I'm looking for is more like a debugger or profiler. It actually runs the program, and then reports if at any time the type annotations deviate from the actual types. (I guess the general case is too costly - think of typechecking a huge list - but for most cases it should be possible.) There are runtime type checkers for python, but the all require modifying the code, and they don't work properly on the main module.
> But as a general rule: You don’t know if there really is a tool in place to check them.
Cool. So you just go and do stupid stuff if no one is looking?
Check it yourself. They're probably there for a reason. If you think they're not needed, then sure, take them out.
> Most of this turned out to be wrong and it threw me off the track during my debugging session.
Here's an idea: Then why don't you fix it. If I see code that's wrong or bad, I go and fix it. It's your responsibility as well
Now maybe try acting like they're anything but type hints (and I'm very glad Guido insisted it is optional and flexible - the last thing the world needed is for Python to get consumed by Java type checking pedantry where the compiler needs you to annotate every single thing)
And every time I do something in Python that would be "impossible" in Java I'm glad I'm not bound by the small-minded type pedantry of Java.
Suppose you start with the function
def foo(bar, baz):
if bar > baz > 0:
return bar / baz
return "bar is not bigger"
Now you start adding type annotations to foo:
def foo(bar: float, baz: float) -> float:
...
def foo(bar: float, baz: float) -> Union[float, str]:
...
Part of the problem is that, in the minds of many, Python == CPython. PyPy, which is a real compiler, is viewed as "nonstandard". For PyPy, type information, although not in type hint form, would be useful. But type hints, as currently defined, are not. See the commentary on type hints at [1].
The takeaway that type hints are worse than doing nothing just doesn't comport with my experience where the type hinting has caught very legitimate subtle bugs one code that was infrequently used. I wonder if the author would make the same argument for TSAN, UBSAN, or ASAN? Coverity? Or other annotation systems that help outside of the compiler? These tools are godsends, but do have false positives if their scope is understood.
Well, there's always Nim. [0]
> ...but it should also not be compiled because then it wouldn’t be as easy to use as Python anymore. Whoops?
Eh, there's Nimscript? [1]
> warnings of IDEs are simple to ignore
This is unusual. In my experience, of codebases I have worked with or have seen, when there are type hints, there are almost all perfectly correct.
Also, you can setup the CI to check also for IDE warnings. For example, we use this script for PyCharm: https://github.com/rwth-i6/returnn/blob/master/tests/pycharm...
The test for PyCharm inspections only passes when there are no warnings.
Although, I have to admit, we explicitly exclude type warnings because here we have a couple of false positives. So in this respect, it actually agrees with the article.
But then we also do code review and there we are strict about having it all correct.
Yes, I see the argument of the article that the typing in Python is not perfect and you can easily fool it if you want, so you cannot 100% trust the types. But given good standard practice, it will only rarely happen that the type is not as expected and typing helps a lot. And IDE type warnings, or mypy checks still are useful tools and catch bugs for you, just not maybe 100% of all typing bugs but still maybe 80% of them or so.
> Isn’t it better to detect at least some errors than to detect none at all?
> You can not be sure that they are correct. As such, you must always treat them as if they were wrong.
I don't get this argument. Isn't this the case for all other code as well? Most code has not been formally verified to work 100% correct. So you assume always all code is wrong? This doesn't make sense.
> They [type hints] waste mental energy when reading code
How? The author even acknowledges that you could just treat them as code comments if you like. By that argument, all code comments waste mental energy?
It isn't that you have to assume the code is wrong (you always have to assume that code is wrong). It's that even though you write type hints and they pass mypy, you still have to assume that the code has type errors. That is, mypy doesn't rigorously check that the program's types match its annotations. It only approximately checks that.
In (some) other languages, the type checker is 100% sound, so if your program passes type checking, you can be completely sure that the types all match. Opportunities for the code to be wrong are thus limited to mismatches between the types and the desired behaviour.
Apparently you can install and setup a billion 3rd party libraries in your code editor and a CI pipeline to get a really good workflow, but that just shows that you need to be an expert in order to use these.
My company runs some kind of static analysis that checks things. And I get 80% of the benefit with 20% of the work. So start small, and don't expect perfection (yet).
It is not called "static typing". It's called "type hinting". Based largely on the name, I wouldn't expect it to error at runtime if you tell me 'hello' is an int. That's just a type hint. I WOULD expect a static analysis to flag is, and mypy indeed does.
Annotating certainly sounds stronger than hinting, but one must dive in and read the PEP to see they in fact do nothing except stash away metadata in a dunder attribute.
My experience with Python is that it slows development speed to a crawl due to dynamic typing. It’s maybe faster to write the first 1000 lines. But after that it becomes slower and more painful. At least in my experience.
Completely disagree. The Python community has very rapidly adapted mypy because of widespread recognition that yes, type information is extremely helpful for any code bases larger than a few files and/or worked on by more than a few developers. Every major Python library I can think of now has mypy stubs available. If you're going to dismiss them as "not Pythonic" you may as well dismiss anything other than Python 2.7 as "not Pythonic".
Like, even if you think that type hints have ugly syntax, prior to them vast most projects just didn't bother with documenting the shape of data, so you had to read source code, guess and experiment. Unless you just don't care about knowing what you're specifically working with (which strikes me as living on the edge), how is it not an ergonomics gain?
def create_user(user):
other_function(user)
Onboarding into a huge, untyped codebase is brutal
Okay, this myth that Pythonic == loosey-goosey duck-type-everything slinging dicts and strings nothing static cause that's too slow, really needs to die.
Pythonic is all about pragmatism and parsimony. If that means not typing anything cause it's a 50 line script, do that. If it means the most efficient way for large dev teams to communicate on sprawling code bases is to use rich types, then do that.
Personally, I use types even in the short scripts, because then I can offload keeping track of what type everything is, instead of having to remember yet another user_dict with whatever keys the producer felt like using at the time. It makes autocomplete faster and I'm less mentally fatigued. Less mental fatigue == more pythonic.
When I'm writing code, and a variable or signature is easy to annotate, I'll annotate it. And guess what? MyPy occasionally catches issues with my types and saves me some work. If a type is very complex, I won't bother spending time annotating. The end result is that I have some code that is annotated and prevents simple errors, and other code that isn't, and I didn't spend much time or effort doing it. To me, that's a clear net win.
I think mypy has to let some potential errors through because otherwise it would spew 1000's of spurious error messages on perfectly good, unannotated legacy code. Something similar happens with the Erlang dialyzer. The same thing applies to tools like Coverity, that aim to find potential bugs in legacy C code.
You could instead imagine a version of mypy that makes no concessions at all to legacy code, and insists that your code be 100% free of type errors, even if that means that some older constructs and styles no longer work. Would that be a good thing? Probably not: we have Haskell for that. Mypy's leaking errors is probably a practical necessity in retrospect. But, I wasn't expecting the leakage, so it surprised and disappointed me when I encountered it. I had thought I was getting something more like Haskell. Mypy still has attractions, but it's less great than I had hoped.
Type hints (with mypy) aren't going to save you from becomming a bad programmer. But they _do_ help you become a better one!
I must be mistaken, but I always thought "type hinting" was a synonym of "optional type annotations". But if you're not using those annotations for actual type checking at some point, what are they good for?
(I am often reminded of my perl days, where something I thought idiomatic 3 days ago, is now completely incomprehensible when I just want to make a minor change)
It's especially bad because, like any comment, it can say one thing but the code may do something different (actually happened to me).
In my experience, it just doesn't work cleanly out of the box like in true statically typed languages, and so I get pushback from my coworkers, who simply can't see the point. And I can't blame them.
If you/your team want to use a statically typed language, then use one. Python is not it.
Usually they can be sold on the path of maximum-reward-for-minimum-effort. Hard to enforce/check type hints are not it (they seem like busywork for no actual payoff). With statically typed languages, the ROI would be different: they would either get the thing to compile, or they wouldn't and leave the job.
This sounds drastic but it's really not: outside the realm of purist conversations between fans of programming languages, people just want to do their job and get it over with. If the tooling is convoluted, has too many rules, or they aren't forced to use it ("...or else"), they just won't.
This is a javascript/python shop, by the way. We cannot choose the languages. We can improve the tooling and train the team with better practices though.
> If you/your team want to use a statically typed language, then use one. Python is not it.
This is a bit of a cop out though, isn't it? A response to criticism of a (possibly) flawed language feature cannot really be "well, use another language". How else will the language improve then?
> it just doesn't work cleanly out of the box like in true statically typed languages
So the response is entirely appropriate. Python is not a statically typed language, of course type hints don't make it behave exactly like a statically typed language.
I meant to say this: because Python's type hints don't work as well as true statically typed languages (based on my own experience), this makes them less useful and also makes them feel more like busywork. Because of this, my coworkers (who have no experience with statically typed languages either, and tend to dislike best practices and features unless they see a clear return of value from them) are hard to convince type hints are worth the time to learn and use them.
Also, let me restate we cannot pick the language. We can just make better or worse use of it.
That question hides the assumption that static languages are always an improvement over dynamic ones, when in reality we should think as different species that have adapted to fit into different environments.
> We can improve the tooling and train the team with better practices though.
"Better practices" are what makes you team more productive, not just blindly copying what other people are doing. If your colleagues really believe that automated tests/type checking are not worth the effort, you are not going to convince them by saying "but so-and-so said otherwise". What you can do is ask for their pain points, and see if the tooling can help with it. You can look at your past burn charts and say "look at this bug here, what caused and why did it take so long to fix? Is this the type of problem that would be easier to solve with static analysis of the code? Look at this refactor that we are planning for next quarter, should we try to increase the test coverage to make sure we have more confidence in the changes?"
Let me give you a twofold answer to this.
1. I do think statically typed languages are generally better than dynamic languages for most use cases (with a few exceptions). I don't expect to convince you or anyone else of this. It may even be the case that I'm mistaken about this, but I made my mind after years of experience with both kinds of languages.
2. The alleged hidden assumption is not truly important. One should always criticize flawed features of any language, static or dynamic, and the answer should never be "well, choose another language". How else will languages improve if nobody is working on addressing their pain points?
> "Better practices" are what makes you team more productive, not just blindly copying what other people are doing. If your colleagues really believe that automated tests/type checking are not worth the effort, you are not going to convince them by saying "but so-and-so said otherwise". What you can do is ask for their pain points, and see if the tooling can help with it.
I'm both nodding in agreement and finding it very hard to think of something I said that made you think I disagreed with this.
Shouldn't then the exercise be to figure out if these use cases are applicable to your situation?
What is your team and company optimizing for?
> How else will languages improve if nobody is working on addressing their pain points?
"See, at my work we need to travel between two islands as fast and cheap as possible. My team is used to high-speed motorboats, but my experience tells me that hydroplanes are faster. I tried putting wings on my motorboats, but it still wasn't as fast. No, we can not buy hydroplanes. No, my team is not licensed to fly. But if no one acknowledges that motorboats are slower, how will they ever be as fast as a hydroplane?
And no, I haven't really looked into the fuel costs of planes vs boats..."
Also, do you honestly expect me to detail what my team and company is optimizing for here? What for? What does it have to do with Python type hinting?
Anyone that has seen the py2 -> py3 debacle will tell you that "let's make python static" is not something that could happen unless you are willing to rewrite the entire ecosystem of libraries and applications, and quite possibly upsetting the majority of current users who were attracted to it in the first place precisely because it is so easy to get started with it.
You can not turn Python into a "fully static" language without changing it so much to the point of making into a different beast. And why should others make all this work to fit into your view of what is "best" when you can just use another language in the first place?
I also cannot choose the language. I'm not in a position to choose languages at my current job; I seldom find myself in that position at any job.
But given that your response to the mypy recommendation was to complain (it's slow, it doesn't catch everything) then it makes it difficult to acknowledge the "valid criticism"... as in: people are working on it, what else do you want?
[0] https://mypy.readthedocs.io/en/stable/existing_code.html
What makes you think I'm not using mypy, or that I didn't read the article you linked to or follow its guidelines (which are very sensible)?
In your mind, is it the only possibility when someone criticizes a tool that they are "using it wrong" (to paraphrase Steve Jobs)? No other possible reason?
> "But given that your response to the mypy recommendation was to complain (it's slow, it doesn't catch everything) then it makes it difficult to acknowledge the "valid criticism"... as in: people are working on it, what else do you want?"
Given that I didn't say or imply that I don't believe there are people working on improving mypy and Python type hints, "what I want" is merely to support the article's assertion that the current version of type hints Python provides is confusing and not very good.
Please, for the love of all that is holy, don't recommend to me again that I switch to a different language. That's not how this works.
What I am having trouble to grasp is: if you understand that Python is not a statically typed language, and if you claim that you do not want to "turn" Python into such, why do you keep conflating type hinting with type checking?
They are two separate things. That is the nature of the game. Do you have criticisms about limitations from mypy, fine, but then you are not talking about issues with type hinting but merely the type checker tool.
It's not my business to help you with your trouble grasping things, but why do you think I'm conflating type hinting with type checking? I specifically asked in this comments section what type hinting meant if not what I thought, and was told by several people "it's just standardized comments", which I then explained was unsatisfying (and other people agreed with me).
> "you are not talking about issues with type hinting but merely the type checker tool"
How about both?
Let me suggest you a more productive use of your time: address the article's complaints as a top-level comment (which I see you haven't yet). I suppose you're more interested in the article's topic than in correcting my alleged misconceptions?
Because the "issues" with type hinting that you are talking about are only based on what you wished it was, not on what it is or what it could become!
Ask yourself this: what about the python's type hinting story you think could be improved without turning python into a statically typed language? What about the python's type checking story that is missing or broken and that is attributable to the language and not to the tool?
> I suppose you're more interested in the article's topic.
Quite frankly, no. My motivation for the conversation now is mostly to see how long is going to take you to realize that you are merely wishing that your motorboat could fly like a hydroplane. It's not about "misconceptions", it's about you complaining about something not meeting your unfounded expectations .
Oh, I see. Well, I have no interest in that kind of conversation.
Goodbye.
What I am trying to say is that perhaps the lesson you could take from this it's to not think that something is "inferior" or "needs fixing" just because it doesn't immediately meet your expectations.
It isn't Python's fault that you have to use at the job even though you don't like it.
It isn't Python's fault that you have to use even though you are absolutely certain that dynamically typed languages are worse.
It isn't Python's fault that type hints are just hints and can not become a full-fledged type system.
It isn't Python's fault that mypy is still not mature enough to work reliably for you.
And it certainly isn't Python's fault that your coworkers don't care about all that as much as you do.
I think there's a lesson here, but it's for you rather than me.
This is tiring. You keep making stuff up for the sake of argument. I never claimed I didn't like Python; in fact I like it. I never claimed it was Python's fault.
I thought you were going to "wait until I realize". Go on waiting, then. Off you go!
Anyway, you also had the chance to respond to other questions I made, but instead you preferred to stick to defensive retorts.
I'm not interested in being "educated". In general, you sound very condescending regardless of your claims that you are not snarky.
It's easier to pick on someone's comments than to make a comment of your own, because that would open you to correction and criticism. I think that's what's going on with you in this case.
Regardless, I suggest you stick to commenting about TFA in the future. It will go better for you. You'll be criticized, but that's par for the course.
> I'm not interested in being "educated" by knowitalls.
From where I am standing, you are trying to get your coworkers to adopt some practice without a strong case for how it would help them or the bottomline, purely out of your belief that statically languages are better. If you think I am being the know-it-all here, fine.
> It's easier to pick on someone's comments than to make a comment of your own, because that would open you to correction and criticism.
Someone else in the thread already pointed out that your original comment and TFA are basically a statement of opinion, not of fact. I do not see what kind of comment we can have except the meta-commentary.
Yes, I can see that you fail to acknowledge the unreasonability of your prodding. Time to change your tack, maybe voice an opinion of your own, on the actual article, and expose yourself to critique?
Maybe drop the condescending tone and try to avoid the cross-examination? It's against the rules on HN.
> I do not see what kind of comment we can have except the meta-commentary.
I can't help you with what you "don't see"; I cannot make you fix your blind spots. I've already told you your meta-commentary is unwanted. I do not welcome it, it's unhelpful, and every single assumption you've made has been wrong (and smug) so far. It's not my job to present my case to you, you're not doing a consultancy here, nor do you demonstrate particular expertise on the subject.
I foresee another pointless reply of your own. I'll let you have the last word.
Bye.
In regards to "type hinting/type checking" in Python, my "opinion" is somewhat similar to automated testing: "use it when it can help increase your confidence about the soundness of your program and its design, but don't rely on it as a guarantee of it being bug-free."
In regards to the article, my opinion is that if you want to claim "disappointment" with a language your expectations should be aligned with the language developers and the general motivations of the community at large. Another opinion is that if the author took the time to understand these underlying motivations there would be no "disappointment" and consequently no article, perhaps?
> expose yourself to critique
Hum, that's interesting. I don't think of "expressing an opinion" as something that "exposes oneself to critique". Maybe I am too accustomed with the idea of "keeping my identity small" and "strong opinions, loosely held", that it's natural to me to differentiate an "attack" on the argument vs the person?
Any chance is this why you are so defensive?
> try to avoid the cross-examination? It's against the rules on HN.
First, not rules but guidelines. Second, you have been responding with passive-aggressive insults and retorts which also makes for unpleasant conversation. Third and most important, there is no cross-examination. I am not challenging your motives or trying to invalidate you as a way to invalidate your opinion. The "prodding" is because I don't think your argument has any merit yet, but perhaps this could change if you provided some underlying reason?
> I've already told you your meta-commentary is unwanted.
It's not lost on me that I when I asked "what you would like to change about python's type hinting without changing the language, and what is wrong about python's type checking that is not just a matter of improving the tool", you completely ignored it and preferred to continue with this grating back-and-forth.
Anyway, you are right when you say it's time to lay this one to rest. I hope the next one gets to be more enlightening to both of us.
You continue nitpicking! You would do better if you followed those "guidelines". Indeed this conversation has been unpleasant, but it's all on you. You have been condescending and insulting. I suggest you focus your energies on better enterprises next time, maybe engage with the article itself instead of picking on others. It will do you good.
> The "prodding" is because I don't think your argument has any merit yet, but perhaps this could change if you provided some underlying reason?
Surely you can see there's nothing to gain by explaining myself to someone who thinks "my argument doesn't have any merit" (and who doesn't think that's insulting) and who keeps misrepresenting everything I say? I just said something, shared by many others, and you felt triggered by the opinion: that's fine, I don't want to convince you of anything, nor do I feel like being "educated" by you.
> Maybe I am too accustomed with the idea of "keeping my identity small" and "strong opinions, loosely held", that it's natural to me to differentiate an "attack" on the argument vs the person?
Well, you haven't done a good job at it, then!
What do you hope to gain, at this point? Are you waiting for me to "see reason"? That's not going to happen. Surely you see sooner or later dang will intervene here and tell us to shut it down?
re: insulting, passive-aggressive behavior: I re-read our exchange from the start. I was polite, even agreed with you sometimes ("I'm nodding in agreement, why do you think I think otherwise") while your tone kept getting more condescending with each reply, failing to accept my agreements and prodding in what I guess you felt was an "educational" tone. Honestly, hand on your heart, do you believe your responses were made in the best possible tone and were the best way to conduct an honest conversation? Do you believe the default position of "this argument has no merit, but maybe if this person explains his team's and company's motivations to my satisfaction I might change my mind" is a mindset that is conducive to amicable conversation? I think, if you're honest, you should admit maybe a tiny bit of misbehavior here.
I also read replies you've made in other articles and you seem more reasonable. I'll chalk it up to you not being able to let go at this point, and so you must bite and nitpick at everything I say. I really think you should let it go, make your peace with not being able to convince me of your point of view, and move on to other enterprises.
> someone who thinks "my argument doesn't have any merit" (and who doesn't think that's insulting)
You forgot the yet. No, it's not nitpicking. Without it, it would be a mere judgmental sentence and it would make sense if you felt dismissed. With it, it is a sign that I am trying to reach for a point of agreement or an understanding.
You are claiming I am the one "biting and nitpicking on everything", but you also could've given a much more charitable interpretation to this, much like you could have done it for the last 6-7 exchanges we had.
And sorry, but I really don't see what is "insulting" about it. Well, at least I don't see how this is insulting if we are to have a rational conversation. This would be insulting only for those who'd be tying their self-image to any of the things being discussed.
> Honestly, hand on your heart, do you believe your responses were made in the best possible tone and were the best way to conduct an honest conversation?
Yes...?
Maybe I am relying too much on the Socratic method, but my questions were mostly to dig in for Truth, not to grieve you. But given the way that old man died, perhaps I should try a different path to not end on the same way as he did.
Oh. My. God.
A Socrates complex.
I give up.
I guess you were right all along, this conversation should've ended two days ago. You win the xkcd 386 today. Good night and good luck.
So, unlike a comment, which does nothing, if you run mypy and get some warnings, you can then do something about it (whether it's just # type: ignore or you realise you made a mistake with allocating to variable new_value versus newvalue/new_valu/etc)
(btw, in case someone objects, some comments DO do something, e.g. golang's godoc examples)
If they are for checking, then my other opinion stands: that they are not very good at it (compared to my experience with statically typed languages, even with Typescript!).
I've tried mypy and pytype, and found both to be unsatisfying and hard to sell to the team (which as I mentioned, don't even like writing tests). If you're thinking "well, that's a culture/professionalism rather than a technical problem", you are absolutely right! But I'm left wondering if a language that was better at static typing would be better because, a- the checking would be mandatory, and b- it would be demonstrably more effective than Python's "optional" checking/hinting. At least it'd remove one variable from the problem. Or maybe not, who knows.
If your team doesn't understand the value of things that are essentially standardized comments exposed through the IDE (even if it wasn't for validation, which is also available with IDE integration), you have very bad teammates.
In my opinion, of course.
I should say that anecdotally I find type hinting very useful when I'm reviewing a PR from a part of the code I'm not intensely familiar with.
Useful, but way less useful than if it was actually checked.
i agree with the author, if the hints can't be trusted, even just once, there's no point littering the code with them, and are actually harmful when trying to debug something using faulty hints.
i am a big fan of static typing but not in python. pick a language that was designed around it. you can't make a duck bark.
I wish this was the case with command line tools I can plug into the build/deploy pipeline. I know they exist, it's just they are unsatisfying and there are lots of cases where they miss stuff that is trivial to catch in statically typed languages.
It's simply not true. Every statically typed language works like I described. Even Typescript works like this. You can have your IDE plugins, but you definitely don't need them to perform type checks. It's not true this requires "a nightmare of 30 command line tools", you just need one: the type checker (built into the language in most statically typed languages, but sometimes split into a separate tool).
Besides, your IDE doesn't live in the build pipeline (in your CI/CD tool). So you cannot rely on it.
A type hint would have answered their question immediately (as they were reading the code).
An unenforced type hint is just like any comment: likely to get out of sync with the code, and a human must do all the work of keeping it up to date.
Python typing is like that. If I say a function takes a List[int], but you know I'm just calling a for loop, you can ignore it and hand me a Set[int]. Maybe it breaks some day, but you're allowed to take that risk, if you have a need.
Do either of these things do anything comments couldn't? Not really. But they're ways of indicating intended semantics without formally documenting your stuff, and when most of what you use the language for is scripts, that's actually pretty helpful. People actually use type hints in a way they didn't with comments, and that's caused a major improvement in code readability. Or at least that's been my experience
You can absolutely look at all this and say python's a crazy, terrible language you never want to touch, but if you've got no choice on language for whatever reason, or you're throwing something together and don't feel like writing `private static final synchronized` forty time, type hints are great.
> I view them as similar to python's "private" functions, which are really just functions starting with an underscore
Good analogy! I wish they were more like "private" class methods, which are prefixed by a double underscore and result in name-mangling: you can still access them from outside if you want, but the code will really look ugly. And you absolutely cannot access them accidentally.
Which is what type checking should be all about, right? Preventing accidental misuse?
> Python typing is like that. If I say a function takes a List[int], but you know I'm just calling a for loop, you can ignore it and hand me a Set[int]. Maybe it breaks some day, but you're allowed to take that risk, if you have a need.
That's what drives me crazy. I come from the statically typed world. This bit of Python's philosophy really clashes with my world view. "These types are just something someone wrote, they may or may not accurately describe the code" seems so wasteful and unhelpful to me...
That’s the kind of the culture shock you get learning dynamic type languages with a static type pov. Type hinting is not really the problem here, but can seem that way because it makes dynamically typed code too superficially similar that it enters the uncanny valley if you treat it like statically typed.
One way that might make this easier is to forget about type hinting entirely at first and learn the language “from scratch”, and add the types back after you get used to write dynamically typed code. Dynamically typed code have its advantage and isn’t bad without type hints (well, at least you have to convince yourself on this, or you’ll never be able to learn a dynamic type language), and the type hints just add back some of the nice things static type provides without compromising dynamic type benefits.
In addition, it makes it harder for me to argue in favor of type annotations with my coworkers. My coworkers come from neither world, static or dynamic; they are learning the ropes. And they just can't see the point. I'm confident I would convince them were this a statically typed language, but with Python I'm lost.
At some point everything in proglang is preferences and design decisions, but for me, there are better and worse tradeoffs to make.
So for that List[int] example, you probably want to take a(n) Iterable[int], Iterator[int], or Collection[int] instead, depending on exactly how you use it.
We should think about whether there's a reason why we want to be writing `private static final synchronized` over and over again, after looking at the state of the software engineer these days.
You seem to be describing actual type checking, which I understand (though in my opinion, mypy is not a satisfying tool for this).
> Despite all that, what I’m hoping for is that someone will come along and tell me that I got it all wrong. “When you do it as follows, the system works: $explanation” Because, you know, I’d really like to have static typing in Python.
I want X. Y (which is designed for somthing other than X) doesn't do X. Thus, I don't like Y.
Yes MyPy and type hinting in general has many limitations, rough edges, and occasionally baffling behavior. Despite all that, I think it is helpful for pretty much any Python project. Type hinting does catch a lot of potential issues and the syntax is very simple. I tend to fill in the types even on a first pass at some code since they help so much in tracking various interactions.
I think the bottom line is that something doesn't have to be great in order to be useful.
> The fact that you can put nonsensical types wherever you want and still get a working program has consequences.
Working is questionable, but nothing in C++ prevents this at runtime either, nothing in JVM prevents this at runtime, static typing is not runtime typing, it should not be, it is not the USP.
> There is an Any type and it renders everything useless
Java has object, C++/C has void*, Go has interface{} - none of these render the type system useless, and Python's type system makes it a lot easier to write correct code than Go's type system or C's type system.
While in python you can mark something as "any" and continue using it as is.
https://stackoverflow.com/questions/69334475/how-to-hint-at-...
I would rather my type checker give me the truth than gloss over it. This issue should be kept open (it is) but they should wait until a good solution is found that doesn't gloss over the discrepancy.
When I started learning PyCharm though, I discovered that type checking can actually work well in Python and it is not a waste of time at all. So I advise people now that unless you're using PyCharm, do not waste any time on type checking Python. In which case, have at it. I love it. PyCharm is really the only way to do type-checking productively right now.
I would recommend to try out if you can, the experience is quite good. I personally find “strict” too strict for the current state of python, but “basic” is already very helpful.
The developer experience is still subpar when compared to Typescript but it’s improving fast (but I mean, typescript with VSCode has one of the best developer experience I’ve ever seen).
Without type hints you don't realise how crazy your data structures get in Python. Trice nested dictionaries getting passed around, or pandas running wild across your code base, and on it goes. Instead with type hints we now have insight and can have meaningful discussions on the data structures flowing through our system.
Our team is yet to adopt mypy into our build systems, but hopefully we will introduce it soon.
That's what the --strict flag is for, which the author even quietly uses in a later example without explaining it. Though it may be too noisy to immediately turn it on in some projects.
As for the example of getting something returned from the network that has a mismatching type, there's no way mypy could guess that. You'd have to add an explicit `isinstance(foo, int)` check or use a runtime type checker.
The author does have a point that people might simply use wrong type hints if they don't have a proper type checker set up though. I've seen so much code with `any` instead of `Any`, `[foo]` instead of `list[foo]`, etc. as type hints. Using something like pre-commit and/or editor plugins to run mypy automatically while editing is probably the best. But it's true that even then, without --strict or equivalent, you can too easily satisfy the type checker by simply annotating something as a `dict` or `list`, without explicitly using `Any`.
Still, I don't think this means we shouldn't use type hints at all. It could definitely be more beginner-friendly, but most of the mentioned issues can be addressed with proper configuration.
Though I agree on its negative impact on the language ergonomics. It removes the joy of writing "scripting language" and as a statically typed language it is far from static-type-native ones.
Now I see it as a tax for compatibility: You enjoy dynamic typing at the beginning of the project, but you have to pay it back once you've grown up to a certain scale.
TypeScript to me feels much more like a native static typed language because it is. Does it help to be a transpiler? Maybe. But I would think that Hejlsberg just has done an astonishingly good work to make it compelling. Python's type annotation is good and completely reasonable, but gradual typing is such a hard problem and being good is probably not enough here.
Now people who use other languages are coming to Python and thinking "Wouldn't it be great if..." and that's how Python the language is going to evolve. But we're going to have to go through this interim phase of awkward adolescence before type annotations mature as a feature.
The thing is, Python is just as compiled as, say Java. Python converts your code to byte code, then executes that bytecode in its VM. These are the same steps Java takes.
So this "whoops" doesn't work for me: "Some language that’s as easy to use as Python and it should be compiled and with good static typing, but it should also not be compiled because then it wouldn’t be as easy to use as Python anymore. Whoops?".
The reason the typing in Python is lax was entirely a choice. Maybe not a good choice, but it certainly has nothing to do with "compiled" vs "interpreted".
Rather, you have to look at it like an attribute based categorization:
Compiled:
- types known at compile time
- limited runtime features like stack traces, introspection
- working closer to primitive data types
Interpreted:
- dynamic, with lots of indirection
- wrapped data types (eg PyObject) vs primitives
- rich runtime with stack traces, introspection, monkey patching
Type hints .... well they help the IDE to autocomplete which is great. They also help you to get the order of parameters in a function/method call correct which is useful if the parameters aren't all the same type!
I converted a medium sized project to type hinting once. The original author wrote no tests and obviously didn't design it to be testable so I was looking for some small way to make it less horrendously unmaintainable. Every test run of the source took hours and there would be some trivial syntax error or whatever right at the end.
Type hints weren't so difficult and I hoped that they would "staunch the bleeding" while I tried to get it into testable shape. In this case they did almost nothing - didn't help me stop a single bug - probably because most types were "str" but that's not the fault of the hinting system as an idea, right? It's just an unprofitable load of work in particular situations.
Static typing fans are always going to be disappointed by Python and it's great because there are other languages for them. For the rest of us we can use them where it turns out to be profitable to us.
Unit tests are, however, by 1000x, more important.
But Python does support type hints, and tools like MyPy do use those type hints to evaluate correctness. It seems to me that Python's type hints are indeed Python's way to check for correctness.
Also, unit tests don't verify correctness. They only serve to check whatever small set of ad-hoc invariant developers put up.
That's why I find type hinting to be relatively high effort for relatively low benefit and choose to use it only where I feel it makes life better. The optionality suits me. It's a nice to have but that's all.
Suppose I wanted to hack something up and run it, just as a new idea? Maybe change a library function that is called in 20 places, but I only want to test it in one call site. Nope! Go forces you to polish up your turd of an idea in all 20 call sites before you can see that your algorithm / model / refactoring is nonsense. If you back out of the idea then all that turd polishing was a waste of time.
The mypy-python workflow, on the other hand, lets me write my nonsense, see that it was a bad idea, then throw it away without eating up more time than I needed to with satisfying all the type checking ceremonial stuff. Much nicer. It’s a linter that follows the “make it work, then make it right, then make it fast” iterative workflow that I’m used to.
Alas, I suppose if I wasn’t an imposter in the programming world, I would be able to dream up my changes and think them through in my head properly before committing them to my text editor. Maybe I wouldn’t even need to do that because my ideas are just great all the time instead of 50/50 between ok and terrible. I’m not smart like that, and it felt like go went out of its way to rub it in.
Maybe I used go wrong. I haven’t touched it in a decade. Is there a “hey ho your types are all out of sync but let’s try and compile anyway” option?
The hacky-developer option I’m after is the ability to cull pieces of code if they do not type check, I guess.
"typed holes" in some programming languages
However I did read a good portion of them and I'm confused...I thought Python doesn't use types everything is an object according to their documentation.
Now for my 2 cents, if you don't care, don't read it... a lot of the comments aren't based on facts but opinions and a lack of understanding of the actual language, funny that I write that after my previous sentence huh? :) . Every language has goods and bads, don't think as someone who as a Python programmer or C programmer, just a programmer and use the best tool for the task you're trying to complete. If you've been programming for awhile you should have used a dozen different languages at some point. Don't focus on what the language can do but understanding the underlying concepts and then you can easily use the one that makes the most sense for that project.
I too am disappointed a bit by the type hints because they are not expressive enough for me, and some natural python constructs are hard to express. And because mypy has bugs I encounter from time to time.
And now, just a few years later- Rust, Go are two of the hottest languages around, TypeScript is eating JavaScript, Python/Ruby have added static typing constructs, C# continues to do well in its niche and even Java of all languages has become somewhat cool again even though being Oracle product.
It's been an amazing transition to watch in action play out in front of my career, I can't even imagine those who have been in the industry for 30-40years who have probably been through a dozen of these cycles.
And maybe a few more decades after that to go back to the dynamic typing of the 1950s, like that of Common Lisp. :)
I would also strongly recommend using Pyright (the default checker in VSCode) over MyPy. It's so much better, and the author is really responsive on Github.
But yeah, in general trying to use type hints in Python is like trying to discuss philosophy in a mental asylum.
The issues with python code bases aren't from a lack of strict typing or type support, but from a lack of a good testing framework.
Python is popular because it's very beginner oriented, so it's the first language a lot of people learn. It's modern BASIC.
Neither of those things mean that writing dynamically typed code is a good idea. The idea that you can write dynamically typed code faster isn't even true once you get past a couple of hundred lines.
And Python being popular because its basic/beginner oriented is a laughable statement. http://highscalability.com/blog/2012/3/26/7-years-of-youtube.... Not to mention that all the bleeding edge ML stuff is Python first.
Language adoption directly depends on how quickly people can build stuff in that language, and its an exponential effect, because with speed of development comes more libraries, which in turn allows other people to build stuff that uses those libraries quicker. And the initial speed of development directly depends on the programmer having to manage less things. Static typing, in real world, is often a hinderance because code bases are dynamic, and having to go and refactor code because data definition changed takes time. And if you have competent programmers that write clean code and a good code review and testing framework, static typing gives you no advantage since the time spent fixing issues due to failing tests will be equivalent to spending time fixing issues with failure to compile.
Ah the classic "I don't make mistakes" delusion.
It's worth noting that nearly every statically-typed language currently in existence has two separate "type systems" - the static type system which is the formal type system offered to the programmer, plus a runtime type system enforced, at minimum, by the processor (e.g. you cannot divide by 0) that is ultimately different from the static type system. The 'question' in most cases is "how closely does the runtime type system match the static type system?" In many if not most languages, the answer is "not very".
Either that or... ? You should probably complete this thought.
I don't think the distinction you've claimed make much sense, but perhaps you had an alternative which you just never explained.
* fork the language again? We just got out of Python2 hell.
* Docstrings, like we had before they were introduced?
They won’t catch all bugs, but with minimal effort they’ll catch some bugs, and if you make an effort to really dig in, it’ll catch most bugs. To me, to accomplish that overnight on a language with millions of existing lines, that’s a big win.
In what way do they lack more teeth than Java with Object, Go with interface{}, C++/C with void*?
Since types are optional in Python, it's necessarily true that an "Any"-like type is prioritized since it's the default. Moreover, cpython doesn't check anything when it bytecode-compiles.
Any-by-default and no-built-in-checking feels pretty defanged to me.
mypy is pretty whiny about types, and sure, if you add Any everywhere then you have just defeated the type checker, but this is no different from putting interface{} everywhere for Go.
> Any-by-default and no-built-in-checking feels pretty defanged to me.
Is running mypy on the code base that much to ask?
Most of the benefit of types goes away unless _everything_ is typed... my libraries, the libraries my libraries use, etc. I should be able to jump around in and out of library code following types and usages thereof in my IDE, making conclusions that are accurate based on what the types say.
When some of your libraries (or indirect dependencies) don't have types, you can no longer ever make any firm conclusions when exploring a complex codebase based on types. I want to change a method in the Widget type, and my IDE says it's used in 4 places. But in reality, that just means 4 or more places, who knows if there are others.
Adding types to a language that doesn't already have them is very hard for this reason. Until many years pass and they're in universal use, they add little value.
Defining such style rules and validations is necessary with almost every language. A codebase where developers are allowed to use any C++ feature or browser API will become very messy as it grows.
I think you mean compile-time and not runtime. There’s not much use in knowing the type at runtime, since it only enables you to print a better error before crashing (“Expected type: x, actual type: y” instead of “AttributeError: 'str' object has no attribute 'append'”).
Types are only truly valuable at compile-time, where they can prevent compilation of invalid code. This is the reason many statically typed languages use type erasure, which removes type information from the compilation output.
In fact, the type-checking phase can happen even earlier than the compilation phase. This is useful e.g. in an IDE, where you care about type errors but not necessarily about running your program.
Also in my experience it can provide static analysis tools more information and provide autocomplete suggestions in a wider range of coding scenarios.
with the exception of some edgy form / upload issues, it's been bulletproof for me
fastapi feels like something the universe wanted to exist and type hints are a natural DSL for it
My biggest gripe is the conflicting warnings between mypy and pyright. Language servers use pyright by default in the editor, but I strongly believe a Python project shouldn't require nodejs as part of its required CI toolchain. So all my projects use mypy in strict mode in CI. Basically this means that mypy is the standard to meet, and that there are persistent warnings in the editor which I have to ignore.
Is that "line" variable a "LineInvoice" or "InvoiceLine" (yes, two different classes) ? Oh, right, both can be used in this place exclusively because one is subset of the other... sigh
Programming in Python worrying about types instead of behavior reminds me of the quote "you can write FORTRAN in any language".
So while correct, your statement is void of any actionable advise
Compare this with Typescript, where by default the compiler will refuse to compile the code when type errors are encountered. But there are other Typescript compilers that are faster and don't typecheck at all, they just strip the syntax.
If you want the strict behavior, you need to prevent the program from running when typechecking fails. That is entirely possible with mypy.
But moving between a TS and Python codebase, I just find TS much easier to work with. Maybe it’s that type bindings are more widely defined in libraries (though typeshed has quite a few now). Or maybe it’s something about TS being easier to define type vars / infer types in my IDE? I can’t really put my finger on it.
I don't know about that. So many C/C++ programs have core dumped in my life and needed to be valgrinded to debug memory issues, or went past the bounds of a pointer, or overwrote the stack, or... whatever.
And you still don't get correctness without unit tests.
I'm not saying types aren't valuable, but they're not a crutch to program correctness like people seem to think they are.
Python type hinting is like moving backwards in time because the amount of time devs take to "hint", takes away the main reason for using python, that is faster development. Might as well code in Java at this point.
a) what types a function/class accepts as arguments and b) what types are returned.
If you don't check it then I agree it's useless. However, working with libraries that are typed is an _insane_ productivity boost over working with untyped libraries.
Typing encourages a large number of bad coding practices. If you can apply typing to a Python program then you are not taking advantage of the dynamic nature of the language to write simpler and shorter code.
In Python world, unit tests cover what is normally handled by typing.
> Typing encourages a large number of bad coding practices.
I'm sorry, what? What exactly is the bad practice here? Not passing random dicts to my coworkers only to surprise them because the key was userid and not user_id? Or wasting time bugging an overworked dev to ask what their crazy 8 levels of dynamic indirection magic are doing, vs just being able to Cmd-B and step through the code path?
> If you can apply typing to a Python program then you are not taking advantage of the dynamic nature of the language to write simpler and shorter code.
Yeah, no, flat out wrong. You can use typing with dynamicism to do magical things. FastApi works a near miracle, generating serializers, deserializers, openapi docs, automatic api deprecation warnings, not to mention type safety.
Properly typed python is faster to write and safely magical.
There are shorter more concise ways to write your programs if you are not using static typing.
If you are not writing your Python programs in the Pythonic way then you are not really using Python correctly and you do not get most of the benefits of the language.
Unit tests exist for catching bugs in Python. Type hints are for documentation not for ensuring code correctness.
Yes, there are libraries that take advantage of type hints, through they could of used other annonations and it's valid to use type hints along with those libraries.
But as soon as you are attempting to prove your program doesn't have bugs via typing rather than unit testing. You have gone wrong.
Typed Python is far more bug prone, why? Because structuring your Python code to support typing means writing considerably more lines of code.
It not the type hints themselves but it's the way you will structure your program at a high level to support them.
The more lines of code the more bugs it has. Bugs are well known to be proportion to the number of lines of source code regardless of language used or whether the language is typed or not.
That does not bear out in practice. In my experience, bugs have almost nothing to do with LoC. A short bit of complex, I/O heavy code with lots of state is going to be way more bug prone, and far harder to unit test correctly, than 10x the LoC of purely functional, well-typed, stateless code.
Mutation, parallelism, IO, and complex state space are the four horsemen of the bugpocalypse.
But in fact, I find functional, well-typed, properly abstracted code ends up with less LoC in the codebase, because it's more reusable and composes well.
SessionInfoType = Union[AuthzQuerySessionInfo, AuthnReqSessionInfo]
Notice how it is doing a union of two objects. Do you know what those 2 objects are? Those are types as well. They have no business use case.
AuthzQuerySessionInfo = TypedDict( "AuthzQuerySessionInfo", {"name_id": Any, "came_from": Any, "issuer": Any, "not_on_or_after": Any, "authz_decision_info": Any,}, total=False, )
AuthnReqSessionInfo = TypedDict( "AuthnReqSessionInfo", { "ava": Union[dict, None], "name_id": Any, "came_from": Any, "issuer": Text, "not_on_or_after": Union[int, bool], "authn_info": list, "session_index": Any, }, total=False, )
RequestCookieType = TypedDict( "RequestCookieType", { "request_id": Text, "came_from": Optional[Text], "public_id": Text, "is_mobile": Union[Literal[True], Text], "c_challenge": Text, "c_challenge_method": Text, }, total=False, )
So now. not only do we have dicts that represent above data, we also have TypedDicts created specially for types.
To rub salt on the wound, the callsite with these types now looks like
session_info = authresp.session_info() session_info = cast(SessionInfoType, session_info)
WTF!! If we have to see a codebase littered with TypedDicts just to satisfy type annotations and then have to cast everything to that type - why not just wrap it in a class and call it a day?
When I see garbage like this, I seriously question the productivity improvement brought about by python type annotations. I certainly end up spending an inordinate amount of time unraveling nested unions of types all over now. Considering switching to a team that uses Golang.
There is probably that stoic school of thought that if you are forced to walk around a construction site without a helmet, it will make you a more thoughtful and better worker, but... come on
I think the traditional view of types is that they're about declaring something to do with the memory layout of a particular variable. Maybe not always directly, but I think most people have this kind of intuition that an int is an int, a string is a string, a struct is a struct (and a particular, explicit one at that). In Java with subtyping, things get a bit more complex, but it's usually fine.
That's generally not a good way to approach Typescript/mypy. Typescript, for example, is absolutely unconcerned with whatever value lives inside a particular variable. It's very possible in Typescript to create types that can't even ever exist at runtime (type brands are a good example of this). The compilation process for Typescript is basically just stripping the types away as if they were comments, and leaving the Javascript behind.
What Typescript is is a slightly more complicated linter that requires annotations. The types you put in are basically just labels. When writing Typescript, your goal is to give every type a label, and then run a program that tells you if all of those labels are compatible with each other. If there is a label mismatch (you've used label "number" but you've done operation "push" on it) then the linter will simply tell you you've done something wrong. Moreover, these annotations can be very loose - annotate something as `any` (or `Any` in Python) and the linter will accept anything done to it. This means that you can always break free of the linter and tell it that you know best (which is true - it is probably impossible to statically validate whether a given Javascript program will throw an error or not, so any linter will have some cases where it prohibits a valid program - the programmer requires the power to override the linter).
Viewing Typescript and mypy in this context helps a lot, I find. It explains why casting doesn't throw runtime errors (a cast is simply a lint annotation saying that a particular expression has a particular label). It explains why JSON isn't validated when it's parsed (you're just starting that the data has a particular label/shape - it's up to you to ensure that that's true, but you can do that check however you like, including not at all if you're very confident in the quality of your data).
This raises the question about which process is better: one where types get transformed into real data structures where all code is kept valid at runtime, or one where the types act only as assertions for a linter validating whether the code ought to remain valid. In my experience, a lot of it depends on how valuable you find the dynamism in languages like Python and JavaScript. If it's very important for you, then traditional typechecking approaches are probably not very viable because no static type system can ever cover all correct Python programs. However, levels plus a linter will resolve most programme correctly, and still give the necessary overrides to allow for other options.
So when it comes to the questions in the article, I think the author would do a lot of good to approach Python type hints from the perspective of a linter that requires annotations.
but for little things, like just function signatures, it's great!
However, the important errors that I make are "kind" errors.
Type systems can verify that you're adding two ints, but won't tell you when you add the number of bananas to the number of oranges. Of course, both numbers contribute to the total number of snacks.
If you're going to have type-hints, you need pseudo-static analysis.
In which case, I fully declare, publicly, that I am very disappointed with numpy, because it doesnt have good coffee making facilities out of the box.
```foo = { 'hello': 'world', 'bar': ['baz'], } ```
Don't put it on the language not to be able to deal with this kind of stuff.
Sadly, dataclasses ignore type hints as well
this is excatly what pydantic was made for. As well as its integration with FastApi.
FastApi, is really the most beautiful way, I have seen. To use the python type system to its fullest.
Before, it went something like this:
1) Install Python (if you don't have it already)
2) Install your favorite text editor
3) Write some Python!
Now, it goes something like this:
1) Install Python
2) Install your favorite IDE
3) Install plugins for your IDE
4) Install black, mypy, flake8 plus some other nonsensical "helper" library
5) Don't forget your poetry, gotta have that
6) Be careful to pick only 3rd party libraries that have good type hints..
7) God help you if you try writing something yourself, let your IDE do that.
Type hints make not only your code more complicated and unreadable. They complicate your entire workflow.
Let's try again:
Python has become a mess. It is no longer a simple productive language to get into like it used to be. Now, you are expected to buy into it's ecosystem, like it is C# or Java.
Well, it is not. It's Python and I am truly sad they are killing it with thousand cuts.
But that's just a symptom of a mature programming language that's used in a variety of complicated scenarios. You don't have to do it all, and no-one is forcing you. You can always write your one-off script without black, flake8, type hinting, or anything of the sort. If it's just your script then no-one will even know.
On the other hand, if you want your code to be part of the larger ecosystem, then you're expected to write it in a conformant way (e.g. pep8 formatting). And that's a good thing!
Haven't used python in years... you have build pipelines now?! Why? Javascript refugees who can't shake their webpack stockholm syndrome?
Eventually you'll get fed up with not knowing what type each argument to an API call needs to be. You'll either get on board with python type hints, or you'll graduate to a typed compiled language, or you'll languish in the trough of python productivity.
Every point in this blog post strikes me as either (1) unaware of the tooling around python typing other than mypy, or (2) a criticism of static-typing-bolted-on-to-a-dynamically-typed-language, rather than Python's hints. Regarding (1), my advise to OP is to try out Pyright, Pydantic, and Typeguard. Pyright, especailly, is amazing and makes the process of working with type hints 2 or 3 times smoother IMO. And, I don't think points that fall under (2) are fair criticisms of type *hints*. They are called hints for a reason.
Otherwise, here's a point-by-point response, either recommending OP checks out tooling, or showing that the point being made is not specific to Python.
> type hints are not binding.
There are projects [0][1] that allow you to enforce type hints at runtime if you so choose.
It's worth mentioning that this is very analogous to how Typescript does it, in that type info is erased completely at runtime.
> Type checking is your job after all, ...[and that] requires maintenance.
There are LSPs like Pyright[2] (pyright specifically is the absolute best, IMO) that report type errors as you code. Again, this is very very similar to typescript.
> There is an Any type and it renders everything useless
I have never seen a static-typing tool that was bolted on to a dynamically typed language, without an `Any` type, including typescript.
> Duck type compatibility of int and float
The author admits that they cannot state why this behavior is problematic, except for saying that it's "ambiguous".
> Most projects need third-party type hints
Again, this is a criticism of all cases where static types are bolted on dynamically typed languages, not Python's implementation specifically.
> Sadly, dataclasses ignore type hints as well
Pydantic[3] is an amazing data parsing library that takes advantage of type hints, and it's interface is a superset of that of dataclasses. What's more, it underpins FastAPI[4], an amazing API-backend framework (with 44K Github stars).
> Type inference and lazy programmers
The argument of this section boils down to using `Any` as a generic argument not being an error by default. This is configurable to be an error both in Pyright[5], and mypy[6].
> Exceptions are not covered [like Java]
I can't find the interview/presentation, but Guido Van Rossum specifically calls out Java's implementation of "exception annotations" as a demonstration of why that is a bad idea, and that it would never happen in Python. I'm not saying Guido's opinion is the absolute truth, but just letting you know that this is an explicit decision, not an unwanted shortcoming.
[0] https://github.com/RussBaz/enforce
[1] https://github.com/agronholm/typeguard
[2] https://github.com/microsoft/pyright
[3] https://pydantic-docs.helpmanual.io
[4] https://github.com/tiangolo/fastapi
[5] https://github.com/microsoft/pyright/blob/main/docs/configur...
[6] https://mypy.readthedocs.io/en/stable/config_file.html#confv...
Domain name checks out :)
Python is still strongly typed.
Me: what type does this argument need to be?
Python: you have to figure that out
Me: I just pass whatever I want?
Python: Oh no. Usually only a single specific type is supported. But you have to guess what it is.
Me: what if I get it wrong?
Python: the program crashes at runtime
Thanks for the hint.
I'd argue if your functions are full of type-assert like guards, _then_ use another language!
Typechecking allows certain errors to be detected at typecheck time rather than at runtime. I don't personally consider that useless. E.g. A mistake I just made. I'm working in a language I don't know too well right now with strict static typing. The file read function takes a handle. If I pass it a string of the file name, rather than the handle result from file open, it just doesn't compile. In most dynamic languages, that error would not be detected until it executes.
Type checking is just a really poor way to ensure program correctness until you start using an exceptionally strongly typed language like Haskell.
You use types in C / C++ / Java because the compiler needs them to work not because it's a good idea.
This claim doesn't make sense to me. Assuming mypy is regularly run as part of CI, why would adding type annotations cause more bugs than leaving it off?