Our journey to type checking 4M lines of Python
blogs.dropbox.com
blogs.dropbox.com
My personal experience with compile time type checking is that the benefits kick in for a single developer project within a week, and for a multi-developer project immediately.
Interesting analogy, but in my experience it doesn't hold up.
If you have a Python typing bug, it will crash your program almost immediately and you can quickly catch it and fix it. In contrast, C boundary checking issues often come up in relatively rare corner cases, and when they do occur, they often do not result in an exception / error especially if the memory is already paged and set to zero.
In production code, boundary checking errors in C occur all the time, but Python type bugs are pretty rare.
People can change unit tests. The benefit of this, is it's a kind of documentation. People can see the tests and know how it _should_ work, or they can not read them and if they create errors, they'll find out the hard way. Unit tests don't make a specification static, but they do reduce chaos.
Python can be as strongly typed as you like it to be. But of course, with Python, instead of insisting on an incoming parameter being of a particular type, the thing to do is Pythonically ask nicely if it can be the type that you want. If you need parameter x to be a float, call float(x), which will raise TypeError or give you a float. And of course, it gives the implementor of the class the option to implement __float__().
With a small amount of effort, you can design all of your __init__() constructors to do that style of validation. Yes, it requires some elbow grease. But you will not be able to pass bad stuff to my __init__ constructors if I don't want you to. And if you want some parameter y to be an instance of Foo, just call Foo(y) and if it is already a Foo, you get the Foo, and if it can be made a Foo, you get a Foo, and if it can't, you get TypeError.
So of course you actually need to exercise the code also, which is where the hypothesis package comes in.
Anyway, I think one of the reasons Python is popular is that it is very consistent about types. Haskel and Rust are also very consistent about types -- most other languages, not so much. Give me one end of the spectrum or the other. The murky middle is suboptimal.
It requires among other things, a unique and toilsome coding style, constructor overloading (which isn't a first class citizen of python), and you only get late runtime validation (which means, for example, you can easily miss type validation on only one side of a conditional).
Something like pytype detects such errors without any code modifications.
But certainly, it doesn’t remove the need for good test coverage. Now, if you want to argue for less testing, that is a different pitch.
Factory methods are usually better, and I believe what the more modern parts of the standard library have tried to use, but they don't work as well for this specific case.
Once you start doing this in python -- beyond any low complexity usage -- you're better off capturing your arguments in their own object and passing this to the function as a single argument. This object can then enforce whatever you want, and your function can now be a class method.
And then you can build a factory for this object that accepts a few args, maybe some keyword parameters. But, once that gets reasonably complex, just rinse and repeat. It's objects, methods and factories all the way down. And unlimited billable hours.
But once you have type validation i don't really see the use case anymore. Common example of how silly this is is functions that have a special case for handling lists of length 1 by allowing the object to be passed in without being part of a list, eg Union[List[Foo], Foo]. Now you need extra code to handle that special case in your function and the type annotation becomes more verbose. Much easier to just enforce List[Foo] as the only way to call your function and let the caller call you function as do_stuff([x]) if they only have a single item x. The IDE will complain instantly if you try to call it as do_stuff(x) so there is no risk to accidentally call it like that.
At runtime. In production. A compiler would catch the problem at build time, never causing an error in production.
And a nice IDE/code editor would catch the problem as you're typing it
Also, error messages in an IDE can be ignored. Compiler errors can’t.
> So of course you actually need to exercise the code also, which is where the hypothesis package comes in.
These are tests you should be doing anyway. Type checking does not excuse poor test coverage.
Another problem with Python is that the culture of testing is definitely not as strong as in Ruby so you won't even get that to compensate the untyped nature of the language.
So the type of almost any class is really more like Struct{WhatYouThinkTheClassIs; ArbitraryDict;} ... and the temptation to misuse this, say for instance if you don't want to add arguments to function_used_lots_of_places(some_object), can be huge. And more generally, this openness means that there may be discrepancies between what's unit tested for a class, and how that class is used in the wild.
The absolute horror it will be to debug this mess when people starts to use it, my god.
That cannot be because of missing types. If you make just 500 lines python unmaintainable you're just a terrible programmer! In it's core it cannot be a type problem. And that is my main point here; there is a hype going on in the dynamic languages to add types, as if it solves everything. IMHO it's not that simple.
To avoid unnecessary explanation about the value of static typing; I am not against, I'm doing c++ and it serves me well.
But I've seen a frustrating trend recently. Where companies on languages like C#/Java. With a massive code base are moving to Python. I often get recruiter calls to lead a new python team, transitioning away from (Java/C#/etc).
The sole reason, is it takes to long to develop a feature in the legacy language. Which is resultant of outdated software and practices. I generally try and steer them towards CI/CD and newer lighter frame works. But someone way up heard python was the hot new thing. Now all critical back-end services must be in python.
To add to this I pushed for type hinting. But type hinting will "slow down developers". Part of the problem to me, at least. Is the person dictating development frameworks, languages, etc. Isn't really a coder, and heard about thing X at some conference.
I like python, but it's an infrastructure quick scripting language. Additionally a lot of the good python devs I know shifted over to tensor flow and machine learning.
For the past week I've been updating a (much shorter) solo app, written in a statically-typed language, and having a bear of a time getting the compiler to stop throwing false type errors at me. Yes, it's the stable release version of the compiler. That has never stopped any compiler from shipping with bugs. These type inference systems can be pretty complex.
Then again, I seem to run into more compiler bugs in general than anyone I know. How do you all do it?
Just a few days back, I had had to refactor one of the core classes used everywhere and mypy told me exactly what to fix. At that point I remember wondering what I'd have done without mypy.
Thanks a ton.
* An API to request files/upload files
* Authentication/encryption (how do we ensure DB employees aren't arbitrarily reading data on servers?)
* Service that shards data and handles multi-region replication
* That entails multiple datacenters (how much code does it take to keep a DC running?)
* Automated backups to cold storage
* Automated restoration + testing of cold storage volumes
* Soft + hard deletion (deleting hot/warm/cold storage volumes reliably)
* Error handling (retries, host errors, network failures, filesystem corruption, finding data when a host dies)
* Fail-safes like blocking requests when hosts can't handle them, shedding load, etc.
I agree that people generally underestimate the lines of code needed to operate systems at scale, but having used dropbox, I'd still say that's excessive. Note, this is just the Python code, not the UI stuff or the golang or Rust. I've been in multiple 1M-10M line codebases at scale and I just cannot fathom with how with their product simplicity (not engineering simplicity) they could be at that size with a language as expressive as Python.
My guess is this is generously counting a lot of forked libraries. Ungenerously, it makes me think there's a lot of NIH syndrome.
It's already production-ready in my opinion, but it still lacks some modern type-system features that you may be used to from other languages. Ironically, for instance, before Protocols were introduced (and they're still not part of Python core), there was no way to express structural subtyping in MyPy (for a duck-typed language!). And I'd argue they've made the same mistake (and a worse one in some ways) with TypedDict, which is a strict/nominal set rather than an a structural one, making it impossible to use on many real-world collections that have key-value pairs that you want to forgo typing for whatever reason.
Still, Python with MyPy is a huge step up from Python without, and mypyc looks interesting as well.
@attr.dataclass
class Person:
name: str
age: intPEP 557 added Data Classes to Python 3.7 that resemble attrs in many ways.
They are the result of the Python community’s wish to have an easier way to write classes in the standard library that doesn’t carry the problems of namedtuples. To that end, attrs and its developers were involved in the PEP process and while we may disagree with some minor decisions that have been made, it’s a fine library and if it stops you from abusing namedtuples, they are a huge win.
Nevertheless, there are still reasons to prefer attrs over Data Classes whose relevancy depends on your circumstances: attrs supports all mainstream Python versions, including CPython 2.7 and PyPy.
Data Classes are intentionally less powerful than attrs. There is a long list of features that were sacrificed for the sake of simplicity and while the most obvious ones are validators, converters, and __slots__, it permeates throughout all APIs.
On the other hand, Data Classes currently do not offer any significant feature that attrs doesn’t already have.
attrs can and will move faster. We are not bound to any release schedules and we have a clear deprecation policy.
One of the reasons to not vendor attrs in the standard library was to not impede attrs’s future developement.
`dataclass` primarily just provides a faster syntax for defining a simple struct-like class. For that alone I've found it tremendously useful.
At the same time, sometimes you do need more functionality around a "struct-like" class, that that's where attrs comes in (e.g. I use dataclasses/attrs to wrap configuration objects, which mostly just hold values but occasionally need a little more logic). Even then, I often end up extending the basic dataclass with some of my own common methods (e.g. serialization).
What are the problems of namedtuples?
This is just what I recall from reading PEPs, the email lists and release notes. My recollection may not be entirely accurate.
But I think it's dead now: https://blog.pyston.org/
I wonder what's too risky about it? Did you try it and something broke? Instagram's MonkeyType seems to think it works ok in production. https://monkeytype.readthedocs.io/en/stable/stores.html
MonkeyType seems to record the answers and then let you decide (via apply) if you want to agree. I’m a little surprised more people didn’t use this approach, but it probably makes for fairly clumsy type sets that the human is better off saying “Eh, I’ll just document it correctly”.
> I was trying to find ways to make it possible to use the
> same programming language for projects ranging from tiny
> scripts to multi-million line sprawling codebases,
> without compromising too much at any point in the
> continuum. An important part of this was the idea of
> gradual growth from an untyped prototype to a battle-
> tested, statically typed product."Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs." - Bill Gates
With this much Python, Dropbox is in a position where integrating MyPy with Bazel is worth the trouble.
Nothing in the article suggests they’re measuring anything but the scale of the problem.
Lines of code is not a good measure of productivity, but it's a relatively decent measure of complexity. Bad developers can write many lines of code that adds little productive functionality, but still increases the overall code-base's complexity.
A 100k LOC code base is probably not 100X more functional than a 1k LOC code base, but it is very roughly 100X more complex.
mypy --help
Disallow the use of the dynamic 'Any' type under certain conditions.
--disallow-any-unimported
Disallow Any types resulting from unfollowed
imports
--disallow-subclassing-any
Disallow subclassing values of type 'Any' when
defining classes (inverse: --allow-subclassing-
any)
--disallow-any-expr Disallow all expressions that have type Any
--disallow-any-decorated Disallow functions that have Any in their
signature after decorator transformation
--disallow-any-explicit Disallow explicit Any in type positions
--disallow-any-generics Disallow usage of generic types that do not
specify explicit type parameters (inverse:
--allow-any-generics)the reason I'm asking is I remember a blog post from earlier this summer announcing the new desktop app which looked very webtech-ish.
I've mostly used the Java, C++ and Go in my career, and it's pretty amazing to me that I can write write safe, readable code (like Go) on top of high-performance libraries (like C++) and step through code on a prod machine with a remote debugger, get live performance metrics with a single CLI command, etc.
And if you started 5 years ago, you're now likely spending half of your time over the next few months/years porting your code from Python 2 to 3. What a disaster.
Sure theres "enough for the world", if you sum everybody, but i bet that even a company of the size of Dropbox would have a hard time hiring everyone they need if the complexity of the language is the same as Rust or C++.
For instance, if you read the Go manifesto to why the language was created, you can read between the lines that C++ getting in the way where enginners were fighting hard to deal with language complexity.
So the same way as Java before them, they created a language for the "average programmer". And you can see how the language is very pragmatic and how they fight to no put more complexity on the language.
By the way, i know its a personal opinion, but i think Rust will not be a great fit for the cloud backend exactlya because of this kind of thing. It will be hard to create codebase as complex as the ones in Java, because there will be much less man-hours available.
Maybe if generations are getting smarter, IDK, but needing to rely on a big head-count of C++ or Rust enginners for any company, i bet they will have a hard time to fill all the positions they need.
Theres much more C++ enginners than Rust right now, and companies have a hard time hiring them (and i bet that for Rust it will be no different giving its complexity).
My understanding was that the push from C++ to Go was largely motivated by long C++ compile/linking times. Go is designed to compile large code-bases much faster than C++. I've experienced this myself, it's pretty frustrating to wait 2-5 minutes for a C++ app to compile / link just so you can debug it for 30 seconds and start all over again.
I mean, how could they know their link times would be that much better?
I know that Pike et al. worked on compilers for Inferno, and the Ken´s C compiler used some tricks to know beforehand they could make it compile and link very fast, but they couldnt know if the language they were designing would be as fast (to compile) as C or Limbo.
They could say confidently: "we could do better", but they would need more pragmatic motives, even to convince Google to support that effort.
And i bet Google was struggling with C++ enginnering, specially within the newcomers on the team.
I think, both things are true, but developer ergonomics play a very important factor into the why of a language and where it fits in the scheme of things.
That's quaint. When I first started working in C++ professionally, we were at +16 hours for a clean build and link of just one of our systems (circa 2004). Over the years, reduced the amount of code (without sacrificing functionality), shared common code (object model) between client and server (went from about 2M LOC to about 250k LOC), reduced the number of external dependencies, introduced precompiled headers on both windows and Linux, and a clean build was down to about 30 minutes. An incremental build and link could be less than 30 seconds, depending on the scope of the change. But, yeah, C++ leaves a lot to be desired in terms of edit, build, test, repeat cycles.
shouldn't be a factor for a company *at that scale*
I'd agree with you re Dropbox, a relatively "enlightened" modern software company. But they are the exception, not the rule. Most companies, big and small hire the cheapest developers they can find. I'm not talking about silicon valley tech companies, but think e-commerce and corporate sites (Nordstrom, General Motors) and old school companies (e.g., Goldman Sachs, power utilities).
These companies don't care about the "right tool for the job", they care about the right people for the job. If the "people" are adept at antiquated technologies, then they'll still pass muster.
Sure, you can hire new talent that knows Rust, but how long for them to pick up the relevant domain knowledge? How long for the existing talent that has the domain knowledge to become equivalently proficient in their current tech stack?
Plus, with a rewrite, you risk falling ill to second system syndrome.
Economically and from a practical engineering standpoint, it seems to me that Dropbox is taking a very pragmatic approach. Gradually make working with the exist code easier rather than throwing the baby out with the bath water.
I'm not hating on Rust, but this fervor to rewrite everything in Rust is obnoxious and often short sighted, in my opinion. If you're talking brand new, green field development, by all means, it should be considered.
Rust crates are still full of the type of bugs that Python libraries killed years ago. Garbage collection overall really does make things easier. And Java still does concurrency better than practically everything else out there.
If you are doing systems programming, go for Rust instead of C++. Maybe even Rust instead of C. But, I would reach for Python, Ruby, Kotlin, Clojure or even modern Java before Rust for most non-Javascript projects.
If you're doing concurrency, you should look at how java.util.concurrent does it. And, then, if you aren't doing it that way, you should start asking really hard questions.
Could you be more specific about what triggered such an enthusiastic description?
Take a good look at things like ConcurrentLinkedDeque or ConcurrentSkipListSet and then look at what guarantees they do and do not make and think about the rationale behind those decisions.
And all you have to do in java is create the collection, hand the reference to multiple threads, and IT JUST WORKS like a normal Java collection.
You do NOT want to be using semaphores and locks directly if you have java.util.concurrent at your disposal. Look at the "Concurrent Collections" section:
Most concurrent Collection implementations (including most Queues) also differ from the usual java.util
conventions in that their Iterators and Spliterators provide weakly consistent rather than fast-fail traversal:
they may proceed concurrently with other operations
they will never throw ConcurrentModificationException
they are guaranteed to traverse elements as they existed upon construction exactly once, and may (but are not guaranteed to) reflect any modifications subsequent to construction.
That library is something I sorely miss when I'm operating in another language.
You should also dive into the code itself. Sometimes you'll look at a function and marvel at how much work it really takes to get something right. And sometimes you'll look at a function and wonder how it could possibly be so simple.
Is the typical implementation of those days structures relying on many small scale mutexes to avoid contention, or RCU techniques ?
I think there are alternative paths when contention starts becoming too high.
It's been a while, but the code is the real reference. You can go back and look at the mailing list archives as well.
That's an interesting claim. What do you mean specifically?
I rarely file bugs on Python packages no matter how obscure my setup is.
Now that Rust stable is actually reasonably solid, I just ignore crates that demand nightly.
As for systems programming, I would argue that Rust still isn't an alternative to C++ in several deployment scenarios that many of us care about, although it should eventually get there.