Practical Python Programming
github.com
github.com
I am sure it has been mentioned before on HN, but seems like a good place to share https://github.com/tuvtran/project-based-learning#python
Not so much about practicality, but I've been getting into puzzles and mazes lately https://www.jamisbuck.org/mazes/ (not yet bought his book, but will)
Happy coding to all :)
My favorite is the one called "A Talk Near the Future of Python (a.k.a., Dave live-codes a WebAssembly Interpreter)" which was talked about on here a while back
Highly recommend that talk.
Side note, this example made me pause and be grateful that I'm no longer professionally writing in python as much as I used to be:
@shares.setter
def shares(self, value):
if not isinstance(value, int):
raise TypeError('Expected int')
self._shares = value
I still use python if I'm doing prototyping, exploring APIs or ideas, etc. But I try not to roll it into production unless it's an inconsequential program.
Running a type checker in Python is a way to avoid runtime type errors, just like in compiled languages, but without a performance benefit.
this can be made prohibited by MyPy's --strict flag [1], and enforced with a mypy git pre-commit hook locally, and with a MyPy CI job.
[1] https://mypy.readthedocs.io/en/stable/command_line.html#cmdo...
They can (as in - there's API for that, the rest is up to a community effort) improve performance of a final program, if type-annotated code is passed through Cython with ``annotation_typing=True`` flag:
http://docs.cython.org/en/latest/src/tutorial/pure.html#stat...
https://github.com/cython/cython/issues/1672#issuecomment-29...
Annotation support was added in 3.0, which isn't really a recent version of Python, having been released about 11.5 years ago.
Also, mypy, while it can use Python 3.x annotations, also supports static checking of Python 2.x code using type comments.
However, support for type annotations, specifically, wasn’t added until Python 3.5 (via PEP 484).
Mypy was actually using python 3.x annotation for type annotations before PEP 484 standardized them brought the stdlib typing module, but with PEP 484 there was a common, language-defined standard baseline for mypy and other efforts.
What I can’t understand is the argument that occasional use of `isinstance` is bad, but also that pervasive nominal type-checking via annotations is good.
If it's user input you should sanitize there. In Python this is often easy because you can usually cast as the type you need e.g. with int() and then raise exception if it cannot.
If it's your code, then you should make sure you are passing in an int if that's what is required. If you type hint the input as int, and somewhere in your code you pass in a string, you will get a warning in the IDE.
I would argue that's a very different time and place than checking for instanceof in the running code and that's why isinstance is bad and type-checking is good.
There are always exceptions but it's bad to write them in examples when teaching code.
Statically-typed languages can guarantee to get this right 100% of the time. Can type-checkers for a highly-dynamic language like Python guarantee the same?
No they don't. They usually provide escape hatches for things typesystem does not cover, so there are cases where typechecker will just trust that you know what you are doing (even if you don't).
But more importantly, you don't need 100% to be useful. For aid in IDE, high precision (with somewhat lacking recall) is good enough. Of course, for refactoring, higher recall, the better (but you could substitute lacking recall with tests, which is suboptimal, but viable).
But it's interesting question on what python/mypy (python typechecker) can actually do. The answer here is it depends on configuration. Mypy with default configuration typechecks only typed code (i.e. functions which have type annotations) so you get guarantees only there. But you can configure it to be more and more strict (checking untyped defs, not allowing untyped code, and more), which increases guarantees you get (and it also increases the number of valid programs that it rejects). You can get in python into really strictly typed code, but you can also hit the wall if you need libraries that does not provide proper type hints (unless you write type hints by yourself).
it doesn't have to be nominal, MyPy supports structural sub-typing through Protocols [1]
When starting a large project, it would be better to skip Python altogether and use a proper statically-typed language.
Type hints only make sense when you’re working on an existing large Python project - just as Guido was doing at Dropbox when he added them to the language. This is because type annotations are the worst-of-both-worlds - they require the verbosity of static typing and provide few of its benefits.
> This is not a course for software engineers on how to write or maintain a one-million line Python application. I don't write programs like that, nor do most companies who use Python, and neither should you.
But otherwise, I feel like trying to choose the language that scales to a million lines sounds like dooming in the beginning itself by over-engineering. I attempted that myself but kept coming back to python because it's fast to code, it's forgiving as hell and its customizable. Once you're used to it's quirks you can breeze through it fast. I'm sure ruby and js have the same as well, and with ts you get better typing, but with the typing you can do with 3.8, I'll argue that python gives the _best_ of both worlds if it's especially mixed with a good IDE like pycharm. If you really want, you can also incorporate pytype or something from the get-go. If my Greenfield project's biggest problem is that python doesn't scale, then I'm a very happy man.
I am willing to entertain the idea that I have been very lucky or that I have some programming mannerism which has caused me to skate by this kind of thing, but I just don't get it.
Case in point, I had written an apartment search website in Perl. Now, at this point I had been pretty disenchanted with Perl as a whole due to the "there's more than one way to do it" culture combining with the "executable line noise" syntax to give rise to a lot of very impenetrable write-once, read-never-again code from others. So, when I did this, I used real, appropriately-named variables, eschewing the convenience of $_, and I made sure each line of code did one thing and only one thing if at all possible. I did not pack a lot into a line. Each line should be obvious. Where I felt that it might be subject to interpretation, I added comments as to what I was doing and why. Each function had is own comment section which discussed what it was for and why, possible room for improvement, and so on.
Eventually the duty passed to another set of hands, a bunch of students who had never seen Perl (the rise of PHP was strong at that point) and in the follow-up, they mentioned that they found it very straightforward to simply re-write each line.
I heard back in the past few months about one of those very large projects I did in a previous job, just keeping touch with people. I asked about one of my personal pet projects. "No, it just keeps running." "It's very obvious where to make any changes." All Python. No type stuff.
I just don't see what I am missing.
At the same time, it is not impossible to write a large program entirely in assembly language, it just requires extreme discipline, even more so than python.
It's just a question of degree.
But when you DO get a type problem, and in any sizable code base, you WILL, it only manifests AT RUNTIME, and possibly at a customer site or in production, rather than manifesting at compile time.
You get used to it somehow. I did C and Java for about 10 years before Python. My first exposure to Python was a Computational Physics class and it was maddening not knowing what types went where. Eventually you realize it doesn't really matter as long as you're disciplined and rely on others who are disciplined enough to document their code well. I haven't had the chance to use mypy in the 7 years I've been using Python (across 3 industries; medical devices, scientific devices, and web) but I would be surprised if it significantly reduced the number of bugs I've seen in practice.
This is especially big for any refactoring work - just yesterday I had to change the format of a config file that gets read by both TS and Python scripts. For ts, I updated the typedef and I instantly got every error in the codebase annotated. For python, I had to manually go through each line, using my own brain as the type checker. My brain is much faster and better at being a “squiggly red line spotter” than a type checker, so I finished my TS work in a few minutes and took probably an hour going through the python, despite the projects being similar in size.
My experience with Python goes like this:
"I have to write a really small script... ok, not so small that I can do it with bash. I know, I'll use Python! This is going to be a small, throwaway utility anyway"
(Some weeks later, it's turned into a large monstrosity and I need to refactor something, and everything breaks because refactoring anything nontrivial written in Python is a dangerous proposition)
"Ouch! Python: never again!"
(Repeat a few weeks later)
I do agree (and have the scars to prove it) that this isn't a panacaea but it does help, a lot.
That being said, one of the worst bugs I had in python was in passing in the wrong types to a constructor function, and my tests didn't catch that. To be fair, Mypy would have, but I hadn't annotated that part. At the end of that debugging situation, I would have gladly killed somone for enforced static types in Python.
That being said, the data model is a work of art, and a core reason why I enjoy coding in Python. It's just a shame that pandas kinda sucks.
- Most Python programs I write start their life as tiny scripts, always with the certainty they'll never grow (Narrator: they always grow). I don't know many people who write tests for their scripts...
- Testing in Python means too much effort in the wrong places. Consider that most refactoring problems would have been caught for free by a language with static typing. I've experienced runtime errors because I changed the return type of a function from a single value to a list (or viceversa). A statically typed language would have let me know of my mistake for free, so that I could focus on tests that really matter.
https://hackage.haskell.org/package/turtle-1.5.19/docs/Turtl...
How do the benefits you get, relate to the difficulty of problems you’re solving when programming?
Eg if the cost of types is increased coupling of code but the trade off benefit is it makes fixing typos easy then that’s a bad trade. You’ve made a hard problem more brittle in exchange for making a trivial problem even easier.
> trivial problem
Type checking is quite non-trivial, especially as the logic and types get more advanced (conditional types, index types, mapped types, etc etc). Not just “typos”.
float multiply(float a, float b) { return a * b; }
Before you reach for the polymorphism or further pollute the universe of types consider that it costs less lines of code to erase types here.
What errors would preserving types help with here? What has more utility, less lines of code spent for the same outcomes or an implementation that exists further along the spectrum of typing in the “at compile time” direction? - I don’t believe that question can be answered but in practice i find less lines of code correlates more strongly with outcomes i care about than the degree of typing applied.
Pretty simple, it helps when you do a refactoring that changes the type of the value passed somewhere from number to number[] and the compiler instantly tells you that that’s illegal.
I did something similar just the other day when I had a shared config file read by both TS scripts and Python scripts and I needed to change the format to support a new feature. On the ts side I updated the typedef and the compiler pointed out to me every area that needed updating, on the Python side I had to spend a good chunk of time manually going through the script to figure out what needed updating.
Typing isn't a strong enough tool to combat that class of problem; you can fix the type signatures while introducing a semantic break. You need tests and if you have tests, what does typing bring? Keep your tests fast, measure and maintain a speed of >250 tests per second and remember to test behaviour not implementation - you won't go far wrong.
IDEs have been performing refactoring changes on our behalf in dynamic languages for years at this point. Don't manually edit when we've never had such powerful tooling available to us
* https://www.jetbrains.com/help/pycharm/product-refactoring-tutorial.html
* https://www.jetbrains.com/help/pycharm/structural-search-and-replace.html
Typing has a time and a place, that is without a doubt but the world is better viewed without the static typing lens permanently affixed in place, your efficacy will increase.
I’d never choose to have myself do more work if it’s not needed and the computer can do it.
Those automated refactorings would Not have helped in the situation I mentioned above.
It’s whether it continues to work tomorrow after you’ve added or removed a behaviour. Tests are the secret sauce that keep the cost of change low.
Refactoring (a commit with zero changes to test code) is the other key pillar of long term codebase health.
Because no matter what problem you have, after 12 minutes of googling, you can pip install foo. The language almost doesn't matter because most programming is via library api.
Projects in the npm ecosystem may be somewhat more likely to.have typings if the project exists, but I find that, for anything other than web frontend, where JS is obviously king, the tool I'm looking for is more likely to actually exist in the python ecosystem.
Automatic stub generation may also be helpful [3]
[1] https://github.com/python/mypy/wiki/Creating-Stubs-For-Pytho...
1: among the most important things here, in addition to the strength of the type system, where mypy also lacks
I don't think that quite explains it. We loved Python at version 1, before pip and PyPI were around - although the "batteries included" standard library played a similar role to some extent. For me it's mostly the syntax and the simplicity.
I understand generics have helped here, but they still don't seem quite there. And ironically, I'm finding ML tasks in python to be something that could really benefit from type hints.
Side note, and this bugs me: if people love it - empirically, they do - and you don't understand why - surely that should bring about some introspection? It seems all-too-often to bring about the opposite reflex.
Badly indented python code doesn't run, which it shouldn't - my understanding and the machine's are different. Well-indented python code has a lot less visual noise than other languages. That alone should give pause for thought. Why would you not want that feature?
> Why would you not want that feature?
Easy. Braces/etc. become basically invisible after using the language for a bit so they don’t bother me whatsoever. However moving code around in Python is always a pain because you must make sure everything is placed at the correct indentation level rather than simply letting the formatter take care of it. I’ve definitely experienced bugs from moved code having its first/last line at the wrong level, and it can be particularly confusing when the last line is at a different indentation than the previous. It’s so much easier to just grab a brace-enclosed block and smack it down somewhere (which also provides a good sanity check that you’ve yanked the entire block and aren’t missing any lines).
> Badly indented python code doesn't run
I wish this were the case. What’s actually the case is that badly indented python code will give you potentially different results than what you expect, which ranges from syntax errors to failed tests to very hard to diagnose bugs.
> Side note, and this bugs me: if people love it - empirically, they do - and you don't understand why - surely that should bring about some introspection?
Side note, and this bugs me: if people hate it - empirically, they do - and you don't understand why - surely that should bring about some introspection?
I know I set myself up for the last line. I can only say I've really made a good faith effort to try to understand the explanations for braces, and none make any sense to me. I've had nearly ten years writing more or less python alongside braced languages (I agree it's not a massive pain), and outside of the REPL - I have literally never seen an IndentationError (err.. that's the same as "doesn't run" to me), or "hard to diagnose bugs". Almost never a TypeError, either. Maybe a dozen or so?
I need to understand, from source code, how the instructions flow. For that, I and essentially all other humans need indentation. I never want the machine to interpret the instruction flow differently to me. I genuinely cannot understand how someone can fail to produce python code to that standard. I trivially can in any language which includes syntax specifically for giving a machine a separate understanding of program flow to humans.
Python’s major typecheckers, which add features faster than the type systems of most major statically-typed languages, already support a more robust type system than many industrially-popular statically-typed languages.
It's not Haskell, or even Scala, but then neither is Go or Java.
Now, obviously, you don't get the performance benefits of type-informed static compilation with Python, but performance isn't the issue most people seem to be discussing here.
For example, let's say you use Mongoengine. Now, you can query a collection by using MyObject.objects.get(...) and it will return a MyObject. You might be able to make that work in Mypy. Now, you use some fancy aggregation feature, passing a dict into Mongoengine and it changes the type of the result you get back. How will you typehint that? I think the only way would be to special case it for all your queries.
Dynamic languages have 'reflection' all over the place. Type systems are to terminate quickly, and if they don't, you are hacking them and they perform terrible (game of life in C++ templates and such).
So even if performance is not the goal, you can't even get the correctness property right.
In Haskell, there are libraries that will generate code from schemas, including migrations. They are typed to various degrees. There also shallow embeddings of the PostgreSQL query language (with types for the returned type, unlike the Mongoengine example before!). I just want to demonstrate that in Haskell, typing is not all-or-nothing either, but it is a spectrum, where a Haskell lib will typically end up being more typed than a Python lib. You could 'simplify' their type signatures (get dynamic typing) with GHC.Generic and you'd get something like what is common practice with Python. But it is pretty much impossible to go the other way.
Isnt this the case any time you use a DSL to query external data? Only sometimes someone else have type-hinted the things for you.
I work in a strongly typed language (F#) that interops with JS, and one of our principles is to do the type checking on the F#--JS boundary and then not have to worry about it again.
For that specific contrived scenario (mongoengine, which I've never used, apparently uses .aggregate for the latter rather than overriding .get) with two stub @overloads of [...].get for typehinting, one covering what it takes and gives in the aggregate case of interest and one for what it takes and gives in the simple case.
1. The Python Programming Language. - An in-depth look at Python programming for those who want to advance their Python programming skills. https://learning.oreilly.com/videos/python-programming-langu...
2. Practical Python Programming. Over 13 years in development and taught to more than 400 in-person groups, you'll learn the fundamentals of Python programming. Now released under a Creative Commons licenses.