Adoption of Mypy for Python type checking: 45% already use it, 40% don't plan to
twitter.com
twitter.com
In general, people who follow him are going to be more "into" Python than someone who just uses it to get their job done, and thus more likely to use optional tooling with a large overhead like mypy.
That will skew the results in favor of mypy in this poll. Though the poll getting on the HN front page will balance that somewhat.
I'd consider static type checker as optional if you are playing around with small or throwaway scripts, but for any half serious project, it should be a requirement.
IME I don't consider it slow, and it's definitely not visibly slower than pylint, flake8, or similar tools for other languages.
Dropbox, which has one of the biggest Python codebases, uses it, and most mypy maintainers including Guido work there.
Guido hasn't worked at Dropbox since 2019. He retired for a bit, but is now in the Developer Division at Microsoft.
I see a lot of sort of 'programming culture' for given languages, practices, even discussion of working conditions and etc that on social media are taken as standard operating procedure. Highly active and even highly knowledgeable folks seem to portray various practices and etc are a given ... that I suspect are actually outliers.
What I would really be interested in, is do library developers use or intend to use mypy? Because if your libs are typed checked and your APIs are well constructed, a lot of that safety will flow to the clients.
Maybe those 40% would love to use it if something was changed or fixed. I think these numbers are frankly amazing, a year from now we reach an inflection point where mypy is used by the majority of some segment of the Python developer ecosystem.
I'm imagining a future 5 years down the road where some poor python developer has to use some "polyfill" solution because they want to import a library which was developed with a type-checking solution which is no longer main-stream, or because the type-checkin solution they're using has breaking changes with respect to previous versions.
I think the type systems are simple enough that this is probably a theoretical concern, and won't come up often in practice.
I left python 3 or 4 years ago and not planning to go back (basically the company I was working for mandated they were not going to ever upgrade to python 3 because of the effort required and thus stopped all migration related projects).
The language is nice, and frameworks such as Django are great, but the mess the ecosystem has become is not something I miss.
Migrating several hundred thousand lines of code bases with few people and many dependencies (some of them not maintained, not ported to 3) are not trivial.
These will stay for years still. Actually it is very sad, that significant new tools are basically useless on 2.x (e.g. vscode: lots of waiting on completion, or jumping to definition - if it works at all).
Take `go fmt` or `cargo fmt`. black is the de facto autoformatter, but it's still an optional dependency. Maybe like virtualenv it'll eventually be integrated into the standard library/distro. That would help. With other stuff, the split you describe could be a real problem, although for the majority of Python users, using mypy and poetry is fine.
> basically the company I was working for mandated they were not going to ever upgrade to python 3 because of the effort required and thus stopped all migration related projects
If you're actually serious, this is intriguing to me. Business-wise, I have never seen the math shake out this way. When we've done sizings/estimated developer hours and ROI, it's always been more economic to slowly migrate to Python 3 vs a rewrite, even if the dev team is also already using and well-versed in another language. If you're able to provide more details, please do. It's so rare to have a high-quality discussion about this.
* Python 2 is dead.
* pip is the most popular in the industry, conda serves a different purpose if your focus is mainly on scientific computing
* what does async vs blocking even mean in this context? all languages have the same problem, just use the libraries that are non-blocking.
* mypy is the static type checker that is the most popular (28.3k repositories, 1.9k packages), if you are using other type checkers then congrats, you will just be the CoffeeScript/Reason user of the JS world, refusing to believe that TypeScript won the mindshare
Maybe Node.js? Although it does have some sync function to do io, most of them are still unblocking, and the ecosystem is unblocking by default too. It is harder to find a blocking library than finding a non-blocking ones.
Otoh, development of all the major type checkers is fairly collaborative. Mypy usually gets features first, but not always.
Instead, the approach seems to be adopting a standardized system of type annotations into the standard library. While a typechecker may expose a library interface, it's main use is as an executable tool, not a library (unlike annotations themselves), so it's not super critical to have one in the standard library. It might be convenient to have it in the standard distribution, but as a practical matter it's not like there is a problem getting at least one of them that does a suitable job with standard type annotations anywhere you can get Python.
Not really. TypeScript is a JS superset (and TypeScript tooling works with unadorned JS, too), but Coffee script/Reason are incompatible languages that happen to compile to JS and have static typing. All of the Python typecheckers are more like TypeScript, except that even the type annotations and much of the type system is also standardized as part of the core language. You can literally use any of them on the same code base, and get largely the same results (some will catch some type errors that others miss, some will flag some correct code as possibly erroneous that others would leave alone, but the differences tend to be minor.)
Officially yes, but in terms of actual use? Not really. Plenty of legacy codebases are still around that don't have the resources to upgrade.
This isn't an issue in 2021. Python 2 reached its EOL over a year ago, and those who own old Python codebases have had known for literally almost 13 years now that Python 2 would be EOL'd when Python 3 came out in 2008.
They're like comments, they're only there for developer convenience and can be completely ignored.
Used correctly - i.e. you’re leaning on structural typing, so you invest your time in Protocols that go “quack” rather than hard-to-test, brittle, nominatively typed Ducks who quack - which incidentally is exactly the path the mypy getting started docs seems to focus on, why?! - that’s just setting people up for failure, i digress.
Mypy, pyright etc are fantastic tools in the toolbox, sure to save time but Cargo culting is a pain to deal with. See for example 2 decades of empty getters / setters in Java “in case we ever want to add some behaviour in future”.
Gradual typing of the structural variety is fantastic, you’ll know if you’re doing it right because you’ll spend surprisingly little time thinking about types, you won’t spend any time wrestling with mypy to convince it of your idea and when it does complain your reaction will be a legitimate “aha, yeah ok fair shout, you got me” not “oh ffs!” :-)
One really cool thing about gradual typing - if you get it wrong and i have to take over your code, i can at least turn it off. One of the reasons i love tests so much - I don’t have to accept yours if they’re a bear to maintain, i can easily bring along my own instead.
He couldn't possible mean javascript or rust. Perhaps golang because of the limited subset of types.
https://en.wikipedia.org/wiki/Type_inference#Types_in_progra...
For data engineering, I haven't had a chance to try this due to lack of language options, but I strongly suspect that structural typing would be much more useful than nominal typing.
It may have to do with the types you're using. When I'm doing EDA, I'm usually zooming past the arrays and dicts as quickly as I can, and making a beeline for something like Pandas that lets me manipulate the data more easily. Pandas is fairly pretty resistant to static typing, and getting over to something that does play nicer with static typing tends to also necessitate a lot of wheel reinvention. Tends to also bring a performance cost, too, since I'm often losing a lot of nice fast multithreaded C++ implementation for a mess of pointer chasing and GIL.
Also, it's not uncommon to not be sure about anything in the data structure you're getting; it's so often that the schema or documentation of that data is wrong or outdated or simply missing, so understanding the properties of the data types is a key part of that exploration. You start with eyeballing the data with your code; making some assumptions, and making code to process and analyze the data that doesn't meet your type assumptions, instead of throwing a runtime error as soon as your assumptions are wrong and conflict with the data.
Exploratory data analysis (and much of data science as a whole) requires you to work with data it is, not data as you'd want or data as it should be; and a tool which can't conveniently work with bad, dirty data is simply not a suitable tool for the task.
By the time you have the structure grokked enough to easily work with it in a statically typed language, you've already got a program that is well-typed and strongly typed. Porting it to a static language would sort of be missing the point, because you certainly aren't going to get any static type checking to help you make sure you've correctly ported the code. And data wrangling code is rarely complex enough that I'm particularly worried about safe refactoring.
And I'm not particularly excited to jump to a static language before I start doing serious analysis, because there still aren't any static languages with top-shelf data analysis packages. Partially, I think, because static typing also becomes a hindrance on this end of things. It's really convenient, for example, to be able to tell your graphing package, "Give me a splot matrix of these five columns," and have it be able to do that. A hypothetical static language with two datatypes would need 32 different overloads of the function for drawing splot matrix, or it would need to operate on a specialized data structure that puts a lot of effort into Greenspunning a limited dynamic typing facility into the language, or alternatively (and this seems to be the more popular option), it would need to simply forego having useful things like splot matrices, and instead just stick to the basics.
Also you could just stick that in as an array or string (or var) to start with
Type inference solves a different problem - it doesn’t help you here. For example
type Money {
amount: Amount,
currency: Currency
}
Money(USD, 10) + Money(EUR, 10) # what type should be inferred here?
A runtime exception maybe?(There’s no “best” answer here, a runtime exception might be the pragmatic choice but i like the Map<Currency, Amount> solution to this).
I wondered if there are any studies that support your assertion? The first few results in Google all refute your idea though, e.g. https://www.researchgate.net/publication/259634489_An_empiri...
Or https://www.ics.uci.edu/~jajones/INF102-S18/readings/23_hane...
And that intuitively makes sense. When i’m adopting types, time saving isn’t really what i’m buying. It’s the opposite - i’m saying “hey, i want to invest more time into this to help ensure correctness” - i.e. i am happy to wrestle with the type system if it means the delegation frees my brain for solving the problem.
And this comes back to what i mentioned earlier about domain complexity - you can use types to help make that easier to manage, often though, it’s better to push harder to eliminate complexity in the first place. Occasionally that’s just not possible, it’s a complex problem domain not another CRUD webapp, in those cases types are fantastic.
It's not about the errors themselves, it's about the way that having a strong type system improves quality of life in general.
For instance, in a duck-typed language, you may have code which runs correctly 99/100 times, but with one specific case it fails due to type issues. These kinds of issues are incredibly frustrating and time-consuming to debug.
Also requiring type designations in function signatures makes code self-documenting. You can just look at the code which someone else wrote, and know exactly what the inputs and outputs are. In a dynamically typed setting, there's often guess work involved in exactly what's accepted by a function, and unless it's very well documented you might have to go into the implementation to see exactly what is happening.
I didn't read the articles you linked in detail, but at least the first I would take with a grain of salt since they use Java as the exemplar for typed programming. Java is notorious for having a cumbersome type system.
In my experience, without types you might get a slightly faster time to first result - but in the long term, once a codebase gets to even medium size, types save you much more time than they cost. Especially since they become automatic after a bit of experience and you don't even think about it anymore.
There's a reason Python and Javascript are trying to add types, and why none of the most interesting languages of the past decade have been weekly typed.
I started off loving dynamic, then i spent years preferring static. Nowadays i don’t actually have too strong an opinion on types (despite what this thread might suggest!), both approaches have value.
These days i place value in something else - less lines of code. Expressive languages like Python & Clojure, these make a lot of sense to me because in my experience nothing correlates stronger with bugs than too many lines of code. By this logic i should be writing in APL but i suspect that goes too far the other direction!
>>> class Money:
... def __init__(self, currency, amount):
... self.m = Counter({currency: amount})
... def __add__(self, other):
... self.m += other.m
...
>>> Money("USD", 10) + Money(10, "EUR”)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<string>", line 5, in __add__
File "/var/containers/Bundle/Application/F8EBA093-47B6-4538-A3A5-553C9C2234FA/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/collections/__init__.py", line 805, in __iadd__
self[elem] += count
TypeError: unsupported operand type(s) for +=: 'int' and 'str'
Even Python yells at me for that.In all seriousness though, how far do such mistakes survive? Not past compile here, but in other cases it could. Could it pass tests? Unlikely. Could it pass casually running the code, not impossible but again seems unlikely.
How many PRs on Github for projects written in dynamically typed languages are things that would be caught by static types? Some, it’s non-zero, it’s not anywhere close to a significant proportion though.
See, for example, the absl::Duration apis in c++.
use std::ops::Add;
fn main() {
let usd = Money {
currency: String::from("USD"),
amount: 10
};
let eur = Money {
currency: String::from("EUR"),
amount: 10
};
dbg!(usd + eur);
}
#[derive(Debug)]
struct Money {
currency: Currency,
amount: Amount
}
impl Add for Money {
type Output = Money;
fn add(self, other: Money) -> Money {
Money {
currency: self.currency,
amount: self.amount + other.amount
}
}
}
type Currency = String;
type Amount = i32; // i know :-)
This outputs 20 USD in rust, it’s a design problem not a types problem.It's glaringly obvious that this code is functionally wrong, in no small part bc of the type information.
Money(“USD”, 10) + Money(“USD”, 5)
This works just fine.We’d need more context about how it’s used to say whether we’ve made a bad trade off with our design.
The error isn’t represented in the types, that’s why it compiles and runs. The compiler can’t see the design error here, which is fair, it’s a design problem and types don’t fix that - people are free to misunderstand the business domain and happily capture the wrong rules in types.
>> fact both have support for erroring on this by default
By default what i’ve done here (omitted a runtime check) doesn’t catch the error.
Code in dynamic languages typically takes advantage of the flexibility to do meta-programming, which can significantly both the amount of code required to complete a task and its complexity.
A lot of modern typed languages have got pretty decent meta programming: rust procedural macros, nim, D, crystal, Zig... Where's the equivalent of serde in python? It's mostly runtime type driven deserialization that is orders of magnitudes slower.
It’s not really a cheat answer - what if you could iterate quickly and launch 5 startup ideas in the time to launch 1 written in Zig but when one of your 5 turns out to be the next Youtube / Instagram / Facebook / [insert other huge traffic site launched on a “slow” backend here] then you have the freedom to glue in some optimised native code in the hot path?
I haven’t gotten to macros yet in my Rust playtime so far but Rust does seem like a really nice complement to Python. Go/Swift strike me as a faster Python - nothing wrong with that, Rust strikes me as a different kind of tool, that appeals. Zig / nim are too bleeding edge for me, when i saw the cool kids start to decry Rust, that was my cue to finally go order the O’Reilly book and dig in (i’ll admit to reading up about comptime to see what the fuss was about - cool idea, i’ll stick with trying Rust for now).
I love Emacs for its extensibility and the capability to make changes to the system and see them immediately. Until programming languages like Mun catch up, I doubt it would be possible to replace Emacs Lisp with a statically typed language and retain the same amount of extensibility. Part of the requirements would be having the ability to replace the implementation of any function in the system at runtime, and that's the kind of thing you have no choice but to design your language from the ground up to include.
I guess what I mean to say is that it's not static languages that make me miss dynamic language features, it's that there's no statically typed language that supports the runtime hotloading that I can accomplish with a few Lua scripts (and has enough community traction to support something like LÖVE 2D).
Steve Yegge wrote an article related to this. He suggests that people who use static type systems are mostly trying to catch errors at compile time, errors which have no relevance after the typechecking is over with and the code is run (in the sense that machine code has no understanding of types). Static and dynamic languages will both ultimately output machine code that does the same thing, but one language incurs a massive penalty to flexibility of change. It's a rant, but it made me think that maybe you could have the type checking to catch errors at the time you're specifically looking for errors, and use the dynamism of the underlying implementation later on.
http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.ht...
But I do disagree with one of your assertions:
> Static and dynamic languages will both ultimately output machine code that does the same thing, but one language incurs a massive penalty to flexibility of change.
In my mind, this is a clear win for static languages. As you have said, it's all machine code in the end - but for static types the compiler can often make more precise decisions about the size of a stack frame, for example, where in the dynamic case it's not so easy. Dynamism doesn't come for free, and probably most of the time we don't need to do fancy type manipulation or dynamic dispatch at runtime, so it doesn't make sense to pay that fixed cost on all my code for something which is done only rarely.
Also I have never really understood the lack of "flexibility" imposed by a type system. Sure if I add a field to a type it might break some code elsewhere in my program, but I don't see that so much as inflexibility as much as the compiler catching a bug I would have been scratching my head over at runtime.
And that is the trap. So far each time I’ve come across someone who’s fallen down the types all the way down hole, they’ve shared a view like this.
In the times where i’ve dug deeper, they were applying a bad unit testing strategy where they aimed for 1:1(+) ratio of methods to unit tests - save that granular level of testing for really specific cases. The heuristic to use is “can i refactor my code without changing a single test?” - if your refactor commit has changes to tests then you’re not refactoring your just changing. Your unit test suite is too tightly coupled and is going to cost a fortune to maintain, even more so in a statically typed language.
The fact is, I really wish there were a way to have the ability to build systems that you can recompile at runtime that had types. The recompilation itself is the "problem," or at least an inconvenience. You have to interrupt your flow in order to test even the smallest change. Yes, this is a selfish programmer desire to make building things easier.
The general consensus is that it's not worth having flexibility over the stability that types enforce, and for most cases I would agree. It would be ridiculous to program critical systems like financial software with the expectation you'd be able to walk right up to it and deploy changes whenever you want. There isn't a need for extensibility with that use case. But there are some cases where extensibility is a key part of the development cycle, like Emacs or Excel or game development. Iteration is important in those cases, seeing that you can make some changes and test them out, "teaching" the program what you want until it's just right. My experience is that this is an optimal development cycle, where I'm not prone to be distracted easily and lose more time than I needed to when I intended to just compile something.
It's just that I'm not seeing why having types means you must use a machine-compiled language, meaning you now have to reboot the program to see your changes. I wish there was a best-of-both-worlds solution that works in the weird way I expect. The closest thing I know of is Teal, which compiles to Lua, but it will be a while before it's production-ready.
I've found that there is no one best typing discipline. Different approaches work better for different problem domains. And I'm currently using Python in so many different problem domains that I find it convenient that Python now allows me to choose the best typing discipline for the job.
It's true that this means that Python isn't the best at any particular way of doing typing. But that also seems appropriate. Being a jack of all trades and master of none is sort of Python's thing nowadays.
And by partial I mean only typing the stuff you end up using, so if you just use one function from some huge, untyped package, just type that one part for now.
The way I see it, you write the stub once, and then the API has less ambiguity and you get type checking.
The only problems I have with them are that they are completely ignored by Python itself - you have to run a separate type checker. That's a fairly stone-aged system. And also I with VSCode had something like Typescript's "no implicit any" so it was more obvious where you had to put types that couldn't be inferred.
What do you expect the Python interpreter to do, exactly? Type annotations are not supposed to affect execution.
In my experience, you may want different type checkers with different configurations for different projects.
Honestly, I don't like the direction Python is taking lately. It used to be easy, approachable, dynamically typed, strongly typed language. Now, syntax is getting polluted and there are definitely more than 2 ways of doing things.
For the language that’s famously supposed to have “one obvious way to do it”, that’s a significant difference of opinion on something so fundamental.
Two Python developers today may well have such enormous differences of opinion on what constitutes proper Python, they could be talking about different languages
“Python is at least three different programming languages at this point.”
https://mobile.twitter.com/dabeaz/status/1373296229711376387
So the % who refuse to use type checking is likely lower than 40%
I would love to use it for personal pytorch projects, but only if I could type-check that tensors are the correct shape, etc. Is that possible?
What I found though, is that 'typed' Python is a great language/dialect in its own right.
There's a confluence of simplicity and ergonomics that I find quite appealing, even as compared to mainstream static typed languages.
I had previously submitted to HN with my thoughts on this:
https://news.ycombinator.com/item?id=25520411
I think the primary difficulty I have seen people face is getting a buy-in from all members of the team. One or two members in every team seem to prefer to write Python the "way it was originally intended" and consider type annotations more of a nuisance. And this ends up hurting adoption.
Sometimes code is being plugged between two highly unstable things. It just needs to work and can't trust that any of the data it receives is valid or good.
I should really be using types here.
Like OO in Python, it feels bolted on. The syntax for generics is bad, at least compared to Java. Typing also exposes how poorly thought out Python's collections are. The hierarchy and methods make sense, but some things are in typing, others in collections.abc. It also falls over when using things like functools.partial.
Then there's the frequent complaint that if you're going to go to the effort to use something typed, you should use something that can take advantage of those types during compilation to optimize the code.
1. If you have a good test suite, mypy adds less value than it costs.
2. If you don't have a good test suite, you really need to get one.
How am I wrong?
> 2. If you don't have a good test suite, you really need to get one.
> How am I wrong?
You're not wrong, per-se, but there is an excluded-middle fallacy here:
If you have a good test suite (that already includes tests for the types of returned values), then mypy may indeed add less value than it costs.
If you don't have a good test suite, you do indeed need to get one.
However.
Mypy is very likely to reduce the cost of adding a good test suite, as it obviates the need for making type assertions in the tests by instead allowing those assertions of various types to be made inline with the code as type annotations.
This makes the remaining functional and unit tests shorter and a lot more readable.
> Surely there’s a real statically typed language with similar or better ergonomics?
Please, if someone here knows of one, let me know!
https://www.youtube.com/watch?v=vM2Zoy4Sxhk&t=2181s
My conclusion was that only a small fraction of projects really used them, there were a lot of projects that had type checks in their code but only in "homeopathic" doses. I started using them for some of my Python projects as well (e.g https://github.com/algoneer/algoneer) and while I find them useful I think they're not as useful as a "real" type system in a fully typed language like Golang. Still, they're very useful for discovering simple mistakes that would only show up in unit testing otherwise.
You can also "misuse" them for other purposes, at the end of the presentation I e.g. show how you can implement software contracts with them. Of course this would wreck a type checker like mypy, so don't do that in your codebase. That's probably also one of my critiques as the annotation syntax can in principle be used for anything, but mypy and other tools are not able to deal with code that does that.