The reason we're stuck on mypy at work is because it's the only type checker that has a plugin for Django that properly manages to type check its crazy runtime generated methods.
I wish more python tooling took the TS approach of "what's in the wild IS the language", as opposed to a "we only typecheck the constructs we think you SHOULD be using".
Or in this case, writing it in Rust...
mypy is written in Python. People have forgotten that Python is really, really slow for CPU-intensive operations. Python's performance may not matter when you're writing web service code and the bottlenecks are database I/O and network calls, but for a tool that's loading up files, parsing into an AST, etc, it's no surprise that Rust/C/even Go would be an order of magnitude or two faster than Python.
uv and ruff have been fantastic for me. ty is definitely not production ready (I see several bizarre issues on a test codebase, such as claiming `datetime.UTC` doesn't exist) but I trust that Astral will match the "crazy reality" of real Python (which I agree, is very crazy).
This is a known issue — we're currently defaulting to a conservative Python version, and `datetime.UTC` really doesn't exist until Python 3.11!
https://docs.python.org/3/library/datetime.html#datetime.UTC
We will probably change the default to "most recent supported Python version", but as mentioned elsewhere, this is very early and we're still working out these kinds of kinks!
But yes, if you have a Python version specified in pyproject.toml, we respect that, and if you have a virtualenv, we can see the Python version that was used to create that. And that's what we use to type-check your code.
The default being discussed here is what we fall back on if that project metadata isn't available.
IMO, a less presumptuous criticism would be phrased like "if you did X then benefits Y would happen", or "if you haven't, consider X", or even (the least presumptuous - make it a conversation!) "have you considered X?", rather than "you should do X".
Currently we default to our oldest supported Python version, in which `datetime.UTC` really doesn't exist! Use `--python-version 3.12` on the CLI, or add a `ty.toml` with e.g.
``` [environment] python-version = "3.12" ```
And we'll find `datetime.UTC`.
We've discussed that this is probably the wrong default, and plan to change it.
If I saw something like "datetime.UTC doesn't exist", I'd immediately think "wait, was that datetime.utc", not "ooh it got added in 3.11, I need to change my Python version"
Is there any big downside to do it the boring way, hardcode a list and compare the error to the list?
There are some extremely CPU-intensive low-level operations that you can easily write in C and expose as a Python API, like what Numpy and Pandas do. You can then write really efficient algorithms in pure Python. As long as those low-level operations are fast, those Python-only algorithms will also be fast.
I don't think this is necessarily "cheating" or "just calling disguised C functions." As an example, you can write an efficient linear regression algorithm with Numpy, even though there's nothing in Numpy that supports linear regression specifically, it's just one of the ways a Python programmer can arrange Numpy's low-level primitives. If you invent some new numerical algorithm to solve some esoteric problem in chemistry, you may be able to implement it efficiently in Python too, even if you're literally the first person ever writing it in any language.
The actual problem is that it's hard for people to get an intuition of which Python operations can be made fast and which can't, AST and file manipulation are sadly in the latter group.
Only if you spend more time on the C implementations than on Python. If you have pure Python loops, you'll be slow. You need quite high-level components and minimal Python glue for it to be fast.
This gives somewhat counterintuitive results where declaring and summing a whole list of integers in memory can be faster than a simple for loop with an iterator.
But yeah writing stuff in a different (compiled) language is often better if that means the python interpreter doesn't need to go through as many steps.
TS might transpile to JS and can always be split into a js and type annotation file but is it's own language developed in tandem with the type check based on a holistisch approach to find how to type check then and then put it into the syntax and type checker.
Thats not true for python at all.
Python types where added as annotations to the language many years ago, but not in a holistic approach but in simplistic approach only adding some fundamental support and then extended bit by bit over the years (and not always consistently).
Furthermore this annotations are not limited to type checking which can confuse a type checker (through Annotated helps a lot, but is also verbose, wonder how long until there is a "Annotated" short syntax e.g. by impl @ on type or similar).
Basically what the type annotation feature was initially intended to be and what it is now differ quite a bit (dump example `list` vs. `List`, `Annotated` etc.).
This is made worse that a bunch of "magic" is deeply rooted in python, e.g. sub-classing `Enum`. Sure you have that in JS too, and it also doesn't work that well in TS (if you don't add annotation on the dynamically produced type).
Lastly TS is structurally typed, which allows handling a bunch of dynamic typing edge cases, while Python is, well, in-between. Duck-typing is (simplified) structural typing but `isinstance` is a common thing in Python and is nominal typing...
So yeah it's a mess in python and to make it worse there bunch of annoyances related to ambiguity or to many ways how to do a thing (e.g. re-exports+private modules, you can do that common coding pattern, but it sucks badly).
Python does support structural typing through protocols introduced in version 3.8. They are documented in https://typing.python.org/en/latest/spec/protocol.html.
As a demo, here is part of https://www.typescriptlang.org/play/typescript/language/stru... translated to Python:
from dataclasses import dataclass
from typing import Protocol
class Globular(Protocol):
diameter: float
class Spherical(Protocol):
diameter: float
# In Python, we need to define concrete classes that implement the protocols.
@dataclass
class Ball:
diameter: float
@dataclass
class Sphere:
diameter: float
ball: Globular = Ball(diameter=10)
sphere: Spherical = Sphere(diameter=20)
# These assignments work because both types structurally conform to the protocols.
sphere = ball
ball = sphere
class Tubular(Protocol):
diameter: float
length: float
@dataclass
class Tube:
diameter: float
length: float
tube: Tubular = Tube(diameter=12, length=3)
tube = ball # Fail type check.
ball = tube # Passes.
Found 1 error.
/scratch/structural.py
/scratch/structural.py:37:8 - error: Type "Ball" is not assignable to declared type "Tubular"
"Ball" is incompatible with protocol "Tubular"
"length" is not present (reportAssignmentType)
1 error, 0 warnings, 0 informations
error: lint:invalid-assignment: Object of type `Ball` is not assignable to `Tubular`
--> structural.py:37:1
|
35 | tube: Tubular = Tube(diameter=12, length=3)
36 |
37 | tube = ball # Fail type check.
| ^^^^
38 | ball = tube # Passes.
|
info: `lint:invalid-assignment` is enabled by default
Found 1 diagnosticThe real difference between structural typing and duck typing is that structural typing requires all of a type's declared members to be present for an object to be considered compatible. Duck typing only requires the members that are actually being accessed to be present.
This is definitely more common in dynamic languages, but I'm not aware of any particular reason why that kind of checking couldn't also be done statically.
from dataclasses import dataclass
from typing import Protocol
class Field(Protocol):
diameter: float
class Property(Protocol):
@property
def diameter(self) -> float: ...
class Ball:
@property
def diameter(self) -> float:
return 1
@dataclass
class Sphere:
diameter: float
ball_field: Field = Ball()
sphere_field: Field = Sphere(diameter=20)
ball_prop: Property = Ball()
sphere_prop: Property = Sphere(diameter=20)
/Users/italo/dev/paper-hypergraph/t.py
/Users/italo/dev/paper-hypergraph/t.py:27:21 - error: Type "Ball" is not assignable to declared type "Field"
"Ball" is incompatible with protocol "Field"
"diameter" is invariant because it is mutable
"diameter" is an incompatible type
"property" is not assignable to "float" (reportAssignmentType)
1 error, 0 warnings, 0 information
It doesn't actually preserve typing on the protocol's methods though
Rust devs in particular are on a bend to replace all other languages by stealth, which is both obviously visible and annoying, because they ignore what they don't know about the ecosystem they choose to target. As cool as some of the tools written for Python in Rust are (ruff, uv) they are not a replacement for Python. They don't even solve some annoying problems that we have workarounds for. Sometimes they create new ones. Case in point is uv, which offers custom Docker images. Hello? A package manager is not supposed to determine the base Docker image or Python version for the project. It's a tool, not even an essential one since we have others, so know your place. As much as I appreciate some of the performance gains I do not appreciate the false narratives spread by some Rust devs about the end of Python/JavaScript/Golang based on the fact that Rust allowed them to introduce faster build tools into other programming languages' build chains. Rust community is quickly evolving into the friends you are embarrassed to have, a bit like any JVM-based language that suddenly has a bunch of Enterprise Java guys showing up to a Kotlin party and telling everyone "we can be like Python too...".
Pydantic being so fast because it's written in Rust is a good thing, you can do crazy dynamic (de-)serializations everywhere with very little performance penalty.
Sorry, but this is just not true. Don't get me wrong, I write typed Python 99% of the time (pyright in strict mode, to be precise), but you can't type check every possible construct in the language. By choosing to write typed Python, you're limiting how much of the language you can use. I don't think that's a bad thing, but it can be a problem for untyped codebases trying to adopt typing.
and how you use the type annotations to indicate a type system is inconsistent and incomplete (e.g. NoneType vs. None for inconsistency and a lot of mess related to mataclasses (e.g. Enum) and supporting type annotations for them for incomplete)
the fact that even today something as fundamental as enums have issues with type checking _which are not just type checker incompetence_ is I think a good way to highlight what mess it is
or that `Annotated[]` was only added in 3.9 and has a ton of visual overhead even through its essential for a lot of clean definitions in modern python code (where for backwards compatibility there is often some other way, which can be de-facto wrongly typed but shouldn't be type linted, have fun type checkers).
Or how make a wrapper function with args and kwargs to pass through?
[1]: https://docs.python.org/3/library/typing.html#typing.datacla...
Also: https://github.com/python/mypy/blob/501a07b45af8e44eda665e53...
Also did you know mypy ignores typing of class decorators? You simply can't return a different type other than type[thisclass].
(And uv and ruff have basically proved that at this point)
which brings us to another python issue, python is quite bad at such huge refactoring even with type checkers
but yeah python is by far the slowest widely used language, and for some use cases you can side step it by placing most hot code in C++/Rust extension modules, (or don't care because you are much much more network latency bound) but a type checker probably doesn't belong into that category
And not directly related, but I wish more python modules did proper checks with Valgrind before shipping.
1. use rr for debugging your binary wheel, you can set up watchpoints, and reverse step/continue from the segfault.
2. compile and run your wheel with sanitizers (ASAN, UBSAN).
I rarely use valgrind so I can't comment on that.
Nah, that's just part of the parade of excuses that comes out any time existing software solutions get smoked by a newcomer in performance, or when existing software gets more slow and bloated.
Here's one of many examples:
https://m.youtube.com/watch?v=GC-0tCy4P1U&pp=0gcJCdgAo7VqN5t...
not because they don't want to or because it's to slow
but because it's not really viable without fully executing module loading in a sandbox, which might seem viable until you realize that you still need to type check `__main__` modules etc. and that its a common trend in python to do configs by loading python modules and grabbing the module locals as keys of the config or loading some things might actually idk. initialize a GPU driver :sob: So it's kinda 100% guaranteed not possible to do fully correct type checking for all project :smh:
But also python is one of the slowest popular languages (and with a large margin to any not also "one of slowest" languages). Only by moving hot code into C++/Rust is it fast, which often is good enough, but a type checker is exactly this kind of software where this approach stops working.