Please do not use Python for tooling
borud.no
borud.no
If we lived in a world in which most tooling (currently written as python scripts) were actually compiled, we'd be seeing better-reasoned essays begging us to write tooling as easily-editable scripts, rather than closed-off executables.
I have had scripts that I needed to edit just slightly almost every time I ran them. Having to involve a compile toolchain every time would have been terrible, and is exactly why I reached for Python in the first place.
I don't doubt the author has felt the pain of changing or missing Python dependencies. It seems likely they don't have a good understanding of pyenv or conda. The solution is not to choose something even worse: it's to write up a README on how to use conda!
And it's true, python-code has a relative high maintenance-cost and deployment is still one of the dominating complains. But I would say this is just the cost that comes with the low entry and fast development.
There are other ways than compiling stuff, some are used today, some others are developing slowly. But at the moment python is really not at the point where long term-availability is a big target.
The point is that it doesn't matter what a theoretical developer could do to lower the "cost of ownership" for their end users, but what is actually being done by actual projects. And in the embedded sphere, the situation is pretty sad. You will have to spend a few weeks every year trying to sort out the tooling.
And it isn't like you the user are supposed to hack the tooling. That's not part of the workflow. Or that it is vitally important that one can do quick changes to the tooling while working on it. It really isn't. It just has to work.
Why should the end user have to care about whether you use pyenv or conda? Or have to figure out how to make them play nicely with some code they didn't write and which they only want to run with a minimum of hassle?
It is kind of like demanding you understand and set the timing advance on your car's engine. Why on earth would you even want to know of it, much less fiddle with it?
Even if the platforms you depend on don't do this, you can do it yourself. It won't fix their bugs, but it will make the development environment more deterministic.
I agree that not all embedded developers should have to work with daily or even monthly. Ideally, all they have to do is run to commands:
./scripts/setup-dev-env
source ./scripts/activate-dev-env
Could python be better in this regard? Yes, absolutely.
I wrote a comment a while back to help out someone else with python packaging. Once you get your mind around it, it's not too bad frankly.
I was a bit unclear in the title (which is what most people seem to have read, and then skimmed the blog posting), but the tooling I'm talking about is things like the `west` tool for Zephyr or the `idf.py` tool for ESP-IDF. Tools that are centrally maintained and distributed. Not the ad-hoc tools people use to do various chores in their codebases.
Both of the tools mentioned are notorious for being extremely brittle. Actually, as we speak, I'm trying to figure out why a project that worked before I left on vacation, and which has been untouched for several weeks, now suddenly doesn't build (because the tooling doesn't run). This is actually par for the course. You have to expect things to break for no obvious reason. My guess? I probably updated some other piece of software and it upgraded a common dependency.
I spend about 10-20% of my time dealing with this sort of nonsense.
Could Python's dependency management be less clunky? Absolutely. But ultimately, when it comes to the tools you mentioned, the vendor is the one responsible for not explaining proper Python dependency management in their documentation, or for not creating an ergonomic wrapper around it.
> It is kind of like demanding you understand and set the timing advance on your car's engine. Why on earth would you even want to know of it, much less fiddle with it?
Developers are not end users a technical understanding and capability is to be expected you aren't talking about the tooling for a pc you are talking about the tooling for a microcontroller.
A more accurate example would be the timing belt manufacture expects the car manufacture to provide their own tooling to put the timing belt on their engine, in the same way a pic manufacture expects you to provide your own tooling.
The end user is whoever buys the device you build with the pic in the same way that the end user of a timing belt is the person who buys the car not the manufacture that makes the car.
I think tooling authors need to look into packaging. Python actually has something similar to JARs, zipapps [0], although I have yet to see them in the wild. PyInstaller is a more well-known option.
[0]: https://docs.python.org/3/library/zipapp.html#creating-stand...
"It is okay to feel provoked by this statement. As pointed out previously: you have probably invested a lot of time in Python. You will be inclined to justify and defend that investment. I would urge you to take some time to think about this and try to calm your urge to come up with counter-arguments. Let it sink in and try to be open to the possibility that this is how many users experience software written in Python."
Did you consider I might have a point and that it might be helpful for people to think about it a bit before just going with their gut reaction?
Instead of reading that sentence above and having the gut reaction of not feeling understood, try and be open to the possibility that you didn't make a very strong argument or perhaps didn't communicate it very well.
So yes, I don't think you communicated your point well. An essay about the brittleness of current embedded systems toolkits which suggested that much of the brittleness could be due to the use of Python in those toolkits, that might have been better-received--but wouldn't have gotten the attention a splashy "Please do not use Python for tooling" did.
Perhaps if you'd spent less time predicting what your essay would provoke, and more time thinking about why your essay might provoke that, you'd have written a better essay.
You keep harping on about the last point. Which makes me think that you really felt that it hit home with you.
Yeah... your communication skill is pretty bad on both ends, it turns out: out and in.
I'm a Go developer who has used Python for scripting, and for teaching my daughter about programming. I don't care to defend Python as a language, but I do care about clear communication, which you continue to lack.
Just last week, I spent a whole afternoon getting a particular repository of Python code running on my laptop, even with the help of virtualenv and pyenv. requirements.txt doesn't tell me which version of Python the code was developed with, and several libraries are only available on certain versions, so I have to play the guessing game first of all. Then, some of the modules don't have binaries available for M1, and I can't build them from source because I don't have x, y, and z tools installed. Then there's always some issue with PYTHONPATH. I ended up having to build the whole Ubuntu docker container and develop inside that.
I love Python, it's not always like this, and it's certainly not only Python, but that experience is something I dread anyway coming in to every new Python project. It feels like DLL hell all over again. I have had a much better experience personally with C# and Rust, but admittedly I had much more solo control over those projects.
https://doc.rust-lang.org/cargo/reference/manifest.html#the-...
tensorflow >=1.12,!=2.1.*
C# for example - I had some compatibility issues between versions 2 and 3.1 of .NET Core, but at least the .csproj tells me what version it's supposed to be built with, and the LangVersion property indicates the version of C#.
Really? In a pinch, use a language that's 90% boilerplate and relies on a mastery of the IDE? It's hard to take this seriously.
Relying on IDE < Benefiting from IDE more than other languages.
Python can be great and so can Java. Different strokes. Different use cases.
Java was mentioned because it offers a way to package all dependencies in a single file. From a user perspective this is preferable to "here is a program, now you have to gather all the pieces to make it run without breaking stuff". Whether one loves/hates Java doesn't really enter into it.
The language itself is somewhat irrelevant for the discussion - the way in which software is distributed and installed is the real point. Python isn't particularly nice for tooling since it offloads a lot of work on the consumer. You download a Python program, then you have to care about how you should run it, which language version it is, make sure the dependencies are downloaded etc. And even if you have gotten it to run today, tomorrow it may not work. For myriad reasons.
Which is roughly equal to that of C, C++, and many other languages, being a tiny constant percentage larger than equivalent code in some “modern” language.
I suggest reading up on Java 18 and some of the upcoming features.
Also no Java please, how do I even install JVM on my system?
Node and JavaScript, Ruby, Perl, Bash are out too of course. Bash sometimes works but what if I'm using a different shell?
I think a lot of the issue is that these languages love to install .directories in your home directory where all your libraries live and it works seamlessly for you.. but no one else.
We use python, perl and bash. Generally people who are more comfortable in one than the other just choose that language.
As for the section on python being antisocial, It makes sense to me. Dependency management is a tough thing for python and javascript. However I would not switch to writing Rust or Go for tooling scripts either.
Though Python has been used for devops scripts in at my job for a long time, and has been very very stable, so I'm not sure why it's causing so much pain for the author.
It's easy to get into a place with Python is a nightmare. I worked at a place full of data scientists who couldn't get anything to work reliably because they installed things with
pip --local
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONIO...
depend on your Python and if they are set wrong and a bit of text gets ingested by the system and spit out by a 'print' your Python will crash. Since there are plenty of 'print'(s) that come in with packages you install with pip the answer 'don't print dirty text' isn't an answer.
On top of that there is the fact that pip's resolving algorithm is incorrect. It can solve simple cases but if you add enough packages it will break down.
One unexpected downside of static languages are dealing with web services written in dynamic languages. As an example, I was writing a Slack app in Go a couple years ago. Some response gets unmarshalled into an easy-to-use struct, but contrary to the documentation, the server has no interest in returning data that can always be unmarshaled into that struct. Sometimes instead of a list of 1 element, the value will just be that one element. There is no type "[]Foo|Foo" in Go, so now you have to write a custom unmarshaling function (or say "fuck it all!" and use map[string]any, at which point you're just writing Javascript). This doesn't cause problems for people using Python or Javascript because neither of those care what's in a dictionary, but statically typed languages do, and you'll have to write extra code to work around that.
I agree that it's annoying to require the users and tool author to have the same runtime and packages installed, though. Python is an incomprehensibly large can of worms here. A python package's dependencies are architecture/platform dependent. To install C packages, you need the exact same C compiler that was used to build Python. Some popular packages have an indirect dependency on a Fortran compiler. It's a pretty big nightmare. You could bundle the runtime with the application like Go does, but the runtime is pretty big and the language is too dynamic to remove the parts you aren't actually going to use in advance, so it's not as easy as a sell.
An option there is that if you're using Python for internal tooling, you bless a particular version once a year, install it on every workstation, and say "sorry, you can't use a different version of Python". That will alleviate a lot of the author's problems, but obviously it's exceedingly politically unpopular.
Anyway, there probably isn't an objective truth here, just personal and organizational preferences here. I agree with the author in that whenever I write some tooling in Python, I regret it almost immediately. But it's working for people.
Is this true? I have never encountered this problem.
My experience is that, since CPython is written in C, you only need to have any C compiler for your platform, because the C compilers will be ABI-compatible. I am not sure about Windows, because Windows has some funny issues with C runtime compatibility. But I have upgraded my Python installation and C compiler separately before without issue.
Python is strongly typed, but definitely not statically.
>>> a = "foo"
>>> type(a)
<class 'str'>
>>> a = 3
>>> type(a)
<class 'int'>
Python is Dynamically typed because:
>>> a = "foo"
>>> a / 3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for /: 'str' and 'int'
Edit: Also for the re-declaration thing, you can see this using id(). Python seems do do some fancy value stuff behind the scenes but you can see the variable changing its identifier in the following example:
>>> a = "foo"
>>> id(a)
140105790230896
>>> a = "foo"
>>> id(a)
140105790230896
>>> a = "bar"
>>> id(a)
140105790230512
>>> a = 1
>>> id(a)
140105793782000
>>> a = "foo"
>>> id(a)
140105790230512
>>> a = "1"
>>> a / 3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for /: 'str' and 'int'
>>> type(a)
<class 'str'>
>>> a = 10
>>> type(a)
<class 'int'>
>>> b = 1
>>> type(b)
<class 'int'>
>>> type(a/b)
<class 'float'>Yes, I've been looking into this lately, and the terms are messy. People tend to use strong to mean "strict in ways I like" and weak to mean "permissive in ways I don't like."
Division is a good example of this ambiguity. It always results in a float, so if you divide two integers, even ones that are evenly divisible, you get a float, so that's kind of a conversion, right? But on the other hand, I don't think a function that was defined as taking two integers and returning a float would be considered an implicit conversion, even if it were overloaded to also accept various combinations of float and integer.
Gary Bernhardt has a good discussion: https://www.destroyallsoftware.com/compendium/types?share_ke...
It's not so big as to be prohibitive though, especially in a dev environment... if this became the norm, I would be a lot happier with using python-based tooling than I am today.
You can get them working locally, but it's easiest when the developers stick with the standard library. Needless to say, virtually nobody does.
The actual image is a tar file, and container runtimes include commands for both exporting and importing. Exceptionally portable. More portable than JAR files and Go binaries, at least.
Nobody is going to use your tool if they have to fetch it from a container registry and have to run it isolated from their filesystem and network, though. OCI images are fine if you have a pet app and you have to ship it to a production server. Not that convenient for interactive tools, though. Consider:
$ docker run -v /tmp/foobar:/tmp/foobar debian:stable ls /tmp/foobar
a.txt
Python is a dynamically, but strongly typed language. JS is dynamic, but weakly typed. Java would be an example of a statically and strongly typed language, and C may be a statically and weakly typed language, but I’m not sure this latter is an apt description (you can cast anything in C and it may work, while this will fail in Java if the types are not compatible)
I was mostly talking about the tooling you find in environments like Zephyr and ESP-IDF. I'd happily accept 500Mb binaries if it meant I never have to spend any time making the tooling work ever again.
Yet his entire article is a justification for his defence of his code preferences.
Moreover his article is vague and has no technical depth, despite his controversial statements/advocacy. E.g. not a single line in the article refers to virtual environments of any kind, which is basic to how Python works and dependency management as part of the (arguably Pythonic) tooling process.
Well, duuuuh.
I kinda get the argument. Nobody wants to spend time maintaining the tools they use to make other stuff, but the tools written in Python by Python developers are likely written that way to be useful, maintainable and in a language they're productive in. Immediate time is an important expense when you're writing tools.
You talk about sunk costs, but sinking time into becoming as proficient in (eg) Rust as Python seems ridiculous. Especially for smaller tools.
And why are we picking on Python? Javascript toolchains eat themselves every 18 months. Yeah Python versions upgrade but we're going to pretend that Node doesn't?
If you don't want a fight, don't pick one. Let developers use their own expertise and experience to decide what they use.
That's why I love it for tools. It's MY job to help my users. For them, I write Java. For ME, I write Python. Because it's python's job to help me.
This is not true for most language toolchains by default as far as I know. Most languages don't produce fully self-contained binaries. The developer have to do extra work to create self-contained binaries, otherwise the users still have to work to get the dependencies necessary.
What makes tooling different from system software in my view is that you're using software on a per-project basis, updating it as the project evolves. The developer of the tooling and the user of the tool (another developer) both have responsibilities. They have to agree on a common platform that the tools can target. The tool author is responsible for documenting explicitly any prerequisites and setup steps and they must make sure the tool doesn't implicitly depend on anything more. The user must make sure they've set up an environment for the project that meets the tools needs.
I've personally found that Python works reasonably well for tooling if I as a tool author follow a few guidelines. I require the users only to have a shell, python3, and python3-venv or miniconda installed on their base system. I provide a setup/activation script that creates/activates a virtualenv or a Conda env in the project directory and makes sure that the packages in the also-included requirements.txt or environment.yaml are installed before the tooling is run.
Since the scripts are provided as part of the tools, the tool author becomes responsible for automating the creation of a working environment for the user of the tool. This process can be automated and reproducible based on a frozen requirements.txt or similar, so most of the brittleness can be eliminated by the tool author.
I don't think any other tool implementation language would provide huge benefits to the users. They would usually still need to install some system-wide prerequisites and use some kind of per-project activation script.
The reason I like Python as a tool author is because it's better than writing shell scripts, and it's still easy to include as source with any kind of project. The standard library -- with the argparse, subprocess, urllib, shutil, etc. modules -- is good enough that for simpler tools no external dependencies (nor any activation script nor requirements.txt) are needed, but familiar for many developers.
Perhaps your problems are due to external dependencies (outside of Python)?
One day I wish someone would train something like copilot to automatically bazel all your dependencies. There’s plenty of training data there and you can validate it by making sure the build flags are the same as the cmake or ./configure …
UPDATE: interestingly, I decided to check out and re-build an ESP-IDF project I haven't built since before the summer and now the build breaks because of wrong version dependencies.
My point was that why on earth should I have to manage all this stuff? I just want tools that work. This tooling doesn't really provide that. It provides me with more work.
I think there's an expectation that top management decides which board gets used and developers just cope with whatever tools come with the board.
Now look at the discussion here. A surprising number of people actually expect users to have to understand the runtime requirements and environment of the tooling and seem to think it is the user's job to learn Python and manage this.
Not a lot of people here "get it". And oh boy do people get offended if you suggest that one may have something to learn from the kind of thinking that went into Arduino.
Python definitely has its strengths, and dominates certain domains -- but no one language is all-purpose, so in my book a warning against using one thing for all the things is decent advice.
It was written in C++ using Qt.
Have you ever tried compiling a Qt program on Windows? It involves signing up for an official Qt developer account to even install qmake.
To the point I had to use an unofficial Qt installer CLI app (aqtinstall) [0] to even install the toolchain to build this little shitty app... which still relied on having several Qt .dll files in the same directory as the .exe to work.
Have you clicked on [0] yet? Well, then guess what programming language aqtinstall uses.
One challenge is that when you work on a team and you know there will be people who have to maintain this stuff further down the road, you have to both ensure that they can make use of the technology (are allowed to by their employer) and that it is documented. I've worked with companies that won't allow Docker for instance (for reasons I was never made aware of).
(A lot of people seem to have gotten really upset about this posting. Most of the comments seem to indicate that a lot of people didn't actually read it and/or understand the context. It wasn't a language shitpost. It was a plea for people to please stop making tooling that just creates a lot of work for other people).
Can confirm, good point.
I'm not upset about your post, I was genuinely curious. To state the obvious, Java JARs have fallen out of favor and containers in fashion. I sincerely think it's a change for the better, but your post raises the interesting point of the parallels.