I would disagree with 1. if you need anything more than shell that starts to become a smell to me. The build/testing process etc should be simple enough to not need anything more.
Me, typically I have found it to be a sign of over-engineering and found no benefits over just using shell script/task runner, as all it should be is plumbing that should be simple enough that a task runner can handle it.
> If it works it works, as long as you can run it locally, it'll be good enough,
Maybe when it is your own personal project "If it works it works" is fine. But when you come to corporate environment there starts to be issues of readability, maintainability, proprietary tooling, additional dependencies etc I have found when people start to over-engineer and use programming languages(like Python).
E.g.
> never_inline 30 minutes ago | parent | prev | next [–]
> Build a CLI in python or whatever which does the same thing as CI, every CI stage should just call its subcommands.
However,
> and sometimes it's an even better idea to keep it in the same language the rest of the project is
I'll agree. Depending on the project's language etc other options might make sense. But personally so far everytime I have come across something not using a task runner it has just been the wrong decision.
Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering" causing similarly sized future issues. But then I also know everyone's history is vastly different, my views are surely shaped by the specific issues I've witnessed (and sometimes contributed to :| ), than anything else.
Strong opinions, loosely held :)
> Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering"
Funnily enough running with the first idea I think is creating a lot of the "over-engineering" I am seeing. Not stopping to consider other simpler solutions or even if the problem needs/is worth solving in the first place.
> Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
I quickly asked Claude to convert one of my open source repos using Make/Nix/Shell -> Python/Nix to see how it would look. It is actually one of the better Python as a task runners I have seen.
* https://github.com/DeveloperC286/clean_git_history/pull/431
While the Python version is not as bad as I have seen previously, I am still struggling to see why you'd want it over Make/Shell.
It introduces more dependencies(Python which I solved via Nix) but others haven't solved this problem and the Python script has dependencies(such as Click for the CLI).
It is less maintainable as it is more code, roughly x3 the amount of the Makefile.
To me the Python code is more verbose and not as simple compared to the Makefile's target so it is less readable as well.
UV scripts are great for this type of workflow
There are even scripts which will install uv in the same file effectively making it just equivalent to ./run-file.py and it would handle all the dependency management the python version management and everything included and would work everywhere
https://paulw.tokyo/standalone-python-script-with-uv/
Personally I end up just downloading uv and so not using the uv download script from this but if I am using something like github action which are more (ephemeral?) I'd just do this.
Something like this can start out simple and can scale much more than the limitations of bash which can be abundant at times
That being said, I still make some shell scripts because executing other applications is first class support in bash but not so much in python but after discovering this I might create some new scripts with python with automated uv because I end up installing uv on many devices anyway (because uv's really good for python)
I am interested in bun-shell as well but that feels way too much bloated and even not used by many so less (AI assistance at times?) and I haven't understood bun shell at the same time too and so bash is superior to it usually
Oh, and later the author suggests the script modify itself after running. What the fuck. Absolutely unacceptable way to deploy software.
Also if this still bothers you, nothing stops you from removing the first x lines of code and having it in another .py file if this feels obnoxious to you
> Oh, and later the author suggests the script modify itself after running. What the fuck. Absolutely unacceptable way to deploy software.
Regarding author suggest its removes itself its because it does still feel clutterish but there is virtually 0 overhead in using/having it still be if you are already using uv or want to use uv
Oh also, (I am not the Author) but I have played extensively with UV and I feel like the script can definitely be changed to install it locally rather than globally.
They themselves mention it as #overkill on their website but even then it is better than whatever github action is
Someday, someone is going to have a really big disaster that comes out of casual getting unauthenticated stuff from somebody else's computer.
I think your reason of worrying is either that A) packages can update and contain malware or B) Uv's installation itself. might have malware if any of A) or B) get hacked
Regarding A) I feel like uv's dependencies can be pinned to a certain date to make them reproducible and this can come of help (https://docs.astral.sh/uv/guides/scripts/#improving-reproduc...)
Regarding B) I feel like they provide attestations via GitHub Artifact Attestations and the script could once again be modified to actually verify it via github attestations and they also provide ghcr artifacts (as such immutability) atleast of docker images and I looked further into it and it seems that you can use github artifacts to upload normal binary files as well so I will probably take a look into seeing if I can do something like this for uv's ghcr
Effectively after A) and B) the trust just ends up being reliant on Github's Microsoft infrastructure (usually) and perhaps python infrastructure which is on fastly
But I feel like this is for cases of extremely sensitive workflows But I feel like I might still take a look at it because security still feels very interesting to me and just because of this discussion, I can see at some pointers of following up on curiosity lol
Anyways would love to continue our discussion and probably update you on trying to make a script which could actually be completely pinned (atleast uv binary instead of just running a shell script from the astral servers in such case)
So previously when I have seen Python used as a task runner I think they used UV to call it. Although I don't think they had as a complete solution as your here auto-installing UV etc.
Although the example you've linked is installing UV if missing, the version is not pinned, I also don't think it is handling missing Python which is not pinned even if installed locally. So you could get different versions on CI vs locally.
While yes you are removing some of the dependencies problems created via using Python over Make/Shell I don't think this completely solves it.
> Something like this can start out simple and can scale much more than the limitations of bash which can be abundant at times
I personally haven't witnessed anytime I would consider the scales to have tipped in favour of Python and I would be concerned if they ever do, as really the task runner etc should be plumbing, so it should be simple.
> That being said, I still make some shell scripts because executing other applications is first class support in bash but not so much in python but after discovering this I might create some new scripts with python with automated uv because I end up installing uv on many devices anyway (because uv's really good for python)
Using Python/UV to do anything more complex than my example PR above?
I copied this from their website (https://docs.astral.sh/uv/guides/scripts/#declaring-script-d...)
uv also respects Python version requirements: example.py
# /// script # requires-python = ">=3.12" # dependencies = [] # ///
# Use some syntax added in Python 3.12 type Point = tuple[float, float] print(Point)
> Using Python/UV to do anything more complex than my example PR above?
I can agree that this might be complex but that complexity has a trade off and of course nothing is shoe fits all but there are times when someone has to manage a complex CI environment and I looked at and there are some CI deterministic options too like invoke etc. and when you combine all of these, I feel like the workflow can definitely be interesting to say the least
Once again, I don't know what really ends up in github actions since I have never really used it properly, I am basing its critiques based on what I've read and what solutions (python came quite frequently) and something recently which I discovered (which was the blog)
The other problem with shell scripting on things like GHA is that it’s really easy to introduce security vulnerabilities by e.g forgetting to quote your variables and letting an uncontrolled input through.
There’s no middle ground between bash and python and a lot of functionality lives in that space.
Maybe I keep making the wrong assumption that everyone is using the same tools the same way and thats why my opinions seem very strong. But I wouldn't even think of trying to "parse a semantic version" in shell, I am treating the shell scripts and task runners as plumbing, I would be handing that of a dedicated tool to action.
Now I’m in agreement with you that this is a bad fit for shell scripting, but it is often pragmatic and expedient. And bc there is a cliff between bash and (say) python, some of the time, you’re going to choose the path of least resistance.
Now scale this out to a small team of engineers all facing the same dumb decision of needing to make some tradeoff when they would much rather be writing application logic. The lack of a ubiquitous and robust intermediate language leads to brittle CI fraught with security vulnerabilities.
While the example I provided is a bit contrived, this behavior isn’t hypothetical. I see it everywhere I’ve worked.
I know it is a made up example, but I am not sure Python would improve the situation. I would be looking at either Docker or SBOMs to try and improve the situation and then hopefully the release process would become a lot simpler.
Meanwhile, if you can stick to the very basics, you can do anything more involved inside a container, where you can be confident that you, your CI environment, and even your less tech-savvy coworkers can all be using the exact same dependencies and execution environment. It eliminates entire classes of build and testing errors.
I know I've mentioned it a few times in this thread, just a very happy user and have found it a really good option for a lot of usage. I'll mostly just use the Deno.* methods or jsr:std for most things at this point, but there's also npm:zx which can help depending on what you're doing.
It also is a decent option for e2e testing regardless of the project language used.
It works okay, better than a lot of other workflows I have seen. But it is a bit slow, a bit cumbersome(for langs like Go or Node.js that want to write to HOME) and I had some issues on my ARM Macbook about no ARM images etc.
I would recommend taking a look at Nix, it is what I switched to.
* It is faster. * Has access to more tools. * Works on ARM, X86 etc.
* https://github.com/DeveloperC286/clean_git_history/commit/f8...
Python is a lot easier to write safely.
I've been using it for most of my local and environment scripting since relatively early on.
> The same Python testing code runs on all three platforms.
I have no objections to Python being used for testing, I use it myself for the end to end tests in my projects. I just don't think Python as a build script/task runner is a good idea, see below where I got Claude to convert one of my open source projects for an example.
I think pwsh is worth exploring. It is cross-platform. It is post-Python and the Python mantra that "~~code~~ scripts are read more often than they are written". It provides a lot of nice tools out of the box. It's built in an "object-oriented" way, resembling Python and owing much to C#. When done well the "object-oriented" way provides a number of benefits over "dumb text pipes" that shells like bash were built on. It is easy to extend with C# and a few other languages, should you need to extend it.
I would consider not dismissing it off hand without trying it just because Microsoft built it and/or that it was for a while Windows-only.
Also, the newer versions aren't included with Windows, which would have been useful – instead Windows includes an incompatible older version that admonishes you to download the new version. But why would you download several hundred megabytes of pwsh when you can equally well download any other language runtime?
Also, it sends "telemetry" to Microsoft by default.
Also, the error handling is just awful, silencing errors by default, requiring several different incantations to fix.
Also, the documentation is vague and useless. And the syntax is ugly.
The new versions aren't included in Windows and the old versions are still in Windows for the exact same reasons of Windows backwards compatibility requirements. But at this point the bootstrap on Windows is as easy as `winget install --id Microsoft.PowerShell`.
The error handling isn't far from the bash defaults, but the magic incantations actually tell you what they do versus the number of bash scripts littered with `set -euxo pipefail` is the exact same as the number of scripts that need an `$ErrorActionPreference = "Stop"` and/or a `$PSNativeCommandUseErrorActionPreference = true`.
I find the documentation less vague and more useful than the average `man` page and the syntax is fine (and better than bash) to me, but I understand how much of that is personal preference and familiarity.
I'm a huge fan of "train as you fight", whatever build tools you have locally should be what's used in CI.
If your CI can do things that you can't do locally: that is a problem.
Probably most of the times when this is an actual problem, is building across many platforms. I'm running Linux x86_64 locally, but some of my deliverables are for macOS and Windows and ARM, and while I could cross-compile for all of them on Linux (macOS was a bitch to get working though), it always felt better to compile on the hardware I'm targeting.
Sometimes there are Windows/macOS-specific failures, and if I couldn't just ssh in and correct/investigate, and instead had to "change > commit > push" in an endless loop, it's possible I'd quite literally would lose my mind.
https://github.com/quickemu-project/quickemu [ Quickly create and run optimised Windows, macOS and Linux virtual machines ]
My laptop is an M1 MacBook Air, and I do have an N100 I could use for Windows... I'd just assume use my fast desktop which even emulated is likely faster and not have to move seats.
Oh btw although there are many primitives which help transferring files between VM's and others by having sshfs etc., one of the things which I enjoyed doing in quickemu is using the beloved piping-server
https://github.com/nwtgck/piping-server Infinitely transfer between every device over pure HTTP with pipes or browsers
The speeds might be slow but I was using it to build simple shell scripts and you can self host it or deploy on cf workers too most likely which is really simple but I haven't done it
But for quick deployments/transfers of binaries/simple files, its great as well. Tauri is meant to be lightweight/produce small binaries so I suppose one can try it but there are other options as well
Piping Serrvers + quickemu felt like a cheatcode to me atleast for more ephemeral vm's based workflow but of course YMMV
Good luck with your project! I tried building a tauri app once for android just out of mere curiosity on linux and it was hell. I didn't know anything about android development but setting up the developer environment was really hard and I think I forgot everything I learnt from that but wish I had made notes or even video documenting the process
Completely agree.
> I'm a huge fan of "train as you fight", whatever build tools you have locally should be what's used in CI.
That is what I am doing, having my GitHub Actions just call the Make targets I am using locally.
> I mean, at some point you are bash calling some other language anyway.
Yes, shell scripts and or task runners(Make, Just, Task etc) are really just plumbing around calling other tools. Which is why it feels like a smell to me when you need something more.
IME this is where all the issues lie. Our CI pipeline can push to a remote container registry, but we can't do this locally. CI uses wildly different caching strategies to local builds, which diverges. Breaking up builds into different steps means that you need to "stash" the output of stages somewhere. If all your CI does is `make test && make deploy` then sure, but when you grow beyond that (my current project takes 45 minutes with a _warm_ cache) you need to diverge, and that's where the problems start.
> With uv, it turns out that it's often faster to omit pre-built wheels from the cache (and instead re-download them from the registry on each run). On the other hand, caching wheels that are built from source tends to be worthwhile, since the wheel building process can be expensive, especially for extension modules.
https://docs.astral.sh/uv/concepts/cache/#caching-in-continu...