You can write pretty straightforward readable bash, just as I'm sure you can write pretty gnarly python. Especially if you're intentionally trying to obfuscate.
But all this relies on one specific thing about the autotools ecosystem - that shipping the generated code is considered normal.
I know of no other build system that does this? It feels weird, like shipping cmake-generated makefiles instead of just generating them yourself, or something like scons or meson being packaged with the tarball instead of requiring an eternal installation.
That's a lot of extra code to review, before you even get to any kind of language differences.
That's why you don't commit auto-generated code. You commit the generating code, and review that.
Same reason we don't stick compiled binaries in our repositories. Binary executables are just auto-generated machine code.
Well, there's your problem. If you have unreviewed code, anything can be snuck in. Doesn't really matter too much where in your system the unreviewed code is.
> It's normal to have uncommitted autogenerated unreadable shell code in the tarball.
You need to review everything that goes into the tarball. Either directly, or indirectly by reviewing the sources it gets built from. (And then making sure that your build process is deterministic, and repeated by a few independent actors to confirm they get the same results bit for bit.)
Something built on python won't cover quite as wide a range of (obsolete?) hardware.
Of course, you could do the detection logic with some autotools-like shenanigans, but then crunch the data (ie run the logic) on a different computer that can run reasonable software.
The detection should all be very small self-contained short pieces of script, that might be gnarly, but only produce something like a boolean or other small amount of data each and don't interact (and that would be enforced by some means, like containers or whatever).
The logic to tie everything together can be more complicated and can have interactions, but should be written in a sane language in a sane style.
On a slightly different note, unless the application is written in python, it grosses me out to think of writing scripts in python. IMHO, if the script is more complex that what bash is good at (my general rule of thumb is do you need a data structure like an array or hash? then don't use bash), then use the same language that the application is written in. It really grosses me out to think of a rails application with scripts written in python. Same with most languages/platforms.
Writing anything in C is a bad idea these days, and requires active justification that only applies in some situations. Essentially, almost no new projects should be done in C.
Re-doing your build system, or writing a build system for a new project, counts as something new, so should probably not be done in C.
In general, I don't think your build (or build system) should necessarily be specified in the same language as most of the rest of your system.
However I can see that if most of your system is written in language X, then you are pretty much guaranteed to have people who are good at X amongst your developers, so there's some natural incentive to use X for the tooling, too.
In any case, I would mostly just advice against coding anything complicated in shell scripts, and to stay away from Make and autotools, too.
There are lots of modern build systems like Shake, Ninja, Bazel, etc that you can pick from. They are all have their pros and cons, just like the different distributed version control systems have their pros and cons; but they are better than autotools and bash and Make, just like almost any distributed version control is better than CVS and SVN etc.
I use bash habitually, and every time I have an inscrutable or non-intuitive command, I pair it with a comment explaining what it does. No exceptions.
I also don't clean up after scripts for debuggability. I will offer an invocation to do the cleanup though after you've ascertained everything worked. Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing.
There was a lot of intentionally obfuscatory measures at play here and tons of weaponization of most conscientious developer's adherence to the principle of least astonishment, violations of homoglyphy (using easy to mistake filenames and mixed conventions), degenerative tool invocations (using sed as cat), excessive use/nesting of tools (awk script for the RC4 decryptor), the tr, and, to crown it all, malicious use of test data!!!
As a tester, nothing makes me angrier!
A pox upon them, and may their treachery be returned upon them 7-fold!
If your hammer is repurposed shoe, it's fair to blame the tools.
A good practice, but not really a defense against malice, because if the expression is inscrutable enough to really need a comment, then it's also inscrutable enough that many people won't notice that the comment is a lie.
This fundamental limitation on info prop will never disappear. There is nothing harder to do than to legit get somebody to actually read code.
1. Languages where obfuscation techniques look a lot like business as usual.
2. Languages where obfuscation techniques look weird compared to business as usual.
The presence or lack of comments in #1 situations won't really help to bridge the gap.
And the other major issue here, is that xz is a basic system unit, often part of a bare bones, ultra basic, no fluff, embedded linux deployment where other higher level languages likely wouldn't be. It makes sense for the tools constituting the build infra to be low dependency.
And yes. In a low dependency state, you have to be familiar with your working units, because by definition, you have fewer of them.
Unironically, if more people weren't cripplingly dependent on luxuries like modern package managers have gotten them accustomed to, this all would have stuck out like a sore thumb, which it still did once people actually looked at the damn thing*.
Also, Python build scripts can be living hell too, full of dancing devils that could be introducing backdoors left and right - just look at your average Conan recipe, particularly for larger/more sensitive libraries, like OpenSSL or libcurl.
And you can't really name such small functions well anyway, not when they're broken down for the sake of being small. Case in point, some build script I saw this week had function like `rename_foo_dll_unit_tests` calling `rename_foo_dll_in_folder` calling `rename_foo_dll` calling `rename_dlls`, a distinct call chain of four non-reused functions that should've been at most two functions.
Are all Python build scripts like that? Not really. It's just a style I've seen repeatedly. The same is the case with inscrutable Bash scripts. I think it speaks more about common practices than the language itself (notwithstanding Bash not really being meant for writing longer programs).
The way it works is that you can provide "recipes", which are Python scripts, that automate the process of collecting source code (usually from a remote Git repository, or a remote source tarball), patching it, making its dependencies and transitive dependencies available, building for specific platform and architecture (via any number of build systems), then packaging up and serving binaries. There's a lot of complexity involved.
Here are the two recipes I mentioned:
libcurl: https://github.com/conan-io/conan-center-index/blob/master/r...
OpenSSL v3: https://github.com/conan-io/conan-center-index/blob/master/r...
Now, for the sake of this thread I want to highlight three things here:
- Conan recipes are usually made by people unaffiliated with the libraries they're packaging;
- The recipes are fully Turing-complete, do a lot of work, have their own bugs - therefore they should really be treated as software comonents themselves, for the purpose of OSS clearing/supply chain verification, except as far as I know, nobody does it;
- The recipes can, and do, patch source code and build scripts. There's supporting infrastruture for this built into Conan, and of course one can also do it by brute-force search and replace. See e.g. ZLib recipe that does it both at the same time:
https://github.com/conan-io/conan-center-index/blob/7b0ac710... -- `_patch_sources` does both direct search-and-replace in source files, and applies the patches from https://github.com/conan-io/conan-center-index/tree/master/r....
Good luck keeping track of what exact code goes into your program, when using Turing-complete "recipe" programs fetched from the Internet, which fetch your libraries from somewhere else on the Internet.
I don’t think these issues would necessarily be solved at all by waving a hand and replacing it with similarly complex build tools. Bazel, for example, can be a daunting tool to fully grasp. Any tool used should be well understood. Easier said than done of course.
And shell has a lot of footguns.
That said, what can help drastically here are well-defined best practices and conventions built into the language which, admittedly, bash really doesn’t have.
- Mutating default arguments to functions, so subsequent calls have different behavior
- Somewhat particular rules around creating references vs copies
- Things that look like lambda captures but aren’t quite
- Bash: installed everywhere you want to work (yes, who actually wants to work on Windows ;-)
- C/C++: when speed/size matters there was no alternative except Assembly until recently
- Javascript: until recently this was the most sane option for client side code on the web (Active X and Java applets existed yes but managed to be even worse.)
- PHP: Low cost hosting, Function-As-A-Service way before that became popular, shared nothing architecture, instant reload for local development bliss
- cannot know which object attributes are private or public (and some classes use settable properties so you can't say "just don't set any attributes on non-dataclass objects")
As long as you apply the same standards to what seems to be everyones darling: Javascript.
Javascript has the same amount of footguns as PHP and Bash but has gotten away with it by being cute (and having a whole menagerie if support tools around it to make it possible for ordinary people to write workable code in it).
(Yes, I am qualified to rant about Javascript BTW. I wrote a working map rendering system with pan and zoom and automatic panning based on GPS location using ECMAScript and SVG back in the spring of 2005. I think roughly half a year before Google Maps became public. Back before all the modern JS tooling existed. All I had was JEdit with syntax highlighting. Perl at least let me put breakpoints in my code even back then.
And yes, I have written more JS since then.)
Now you can at least just stick to React and TypeScript and bundle it using Webpack and have months of relative sanity between each time you have to throw something out and replace it.
-> REPL essentially for free (the language IS a REPL)
-> enormous installed base
-> No compilation (well, unless you have something--like autotools--using shell as essentially a transpilation target)
-> No need for "libraries" in most cases: the ordinary CLI that $vendor already ships can be used right away, with no need for a custom SDK for whatever "real" language you would otherwise be using. For example, if you are already familiar with the "aws" CLI program, it's trivial to treat it as an "API" for a quick shellscript instead of needing to dig into the boto3 docs to do something equivalent the "right" way.
-> Pretty good integration with standard *nix facilities (redirecting STD{ERR,OUT}, checking existence of files/pipes, checking/setting exit codes, etc.)
I might agree with "it's not hard to learn it", but I don't agree with "it's not hard to remember it".
I really don't understand this point, its a script language, how old is it doesn't make any difference. I've come accross some Powershell scripts that were unreadable down to its verbosity with certain things, and if you don't already know all the flags and options for it, it's hopeless to try and understand.
Both serve a purpose, neither are 'evil'.
When you use long options in bash (scripts), it becomes very readable, but it's not a widespread practice. I always use long options while writing scripts.
Consider these two examples, which is very straightforward:
- curl -fsSL $URL | bash
- curl --fail --silent --show-error --location $URL | bash
The second one almost "talks you through".
It's true that Bash and Perl has one of the most contractible syntax around, but it's not impossible to make it more understandable.
However, these parts of codebases are considered "supportive" and treated as second class citizens, and never receives the same love core parts of the codebases enjoy. That's a big mistake IMO.
When you make something more readable all around, hiding things becomes harder exponentially.
I challenge you to find a single easily readable >100 line bash script and link it here (I do think small scripts can be fine).
I may add more if I have time today.
Who says I blindly curl2sh these scripts? I download them, read them and understand what they do before running them actually.
Long options make obscure command lines readable and understandable, actually, and if you vet them, you understand what they are doing.
Short options make things harder because I need to open the man of the command to understand what it is doing.
You don't have to do everything blindly as the quick start documents outline.
WTFing at shell scripts is normal.
"WTFing". This is brilliant.
This isn’t just because I’m more familiar with Python. I don’t even think it’s the main reason. It’s just that Python is more likely to be able to be read in a ‘natural language’ sort of way. It’s better at doing what it says on the tin. It’s more able to be read via pure intuition by a programmer that’s not familiar with Python specifically.
In bash land? “What the hell is [[?”
And yes, I could come up with 20 ways off the top of my head that Python is a far-from-perfect language.
And I’m not even saying that Python is the right tool for the job here. Maybe we’re better off with one of the many more modern attempts at a shell language. The main thing is that we should acknowledge that bash has overstayed its welcome in many of the areas in which it’s still used.
The reality is that we as a society have made meaningful progress as far as designing readable languages goes. And we LITERALLY have many orders of magnitude more resources to make that happen. That’s something to feel good about. It’s unreadable to continue to mischaracterise some sysadmin greybeard’s’ familiarity with bash as an indication that it is in any way intuitive or readable.
Like, sheesh, now we all sound like C developers trying to justify an absurdly footgun-laden standard library just because we happen to know the right secret incantations, or think that we know them, anyway. But now this is definitely becoming a religious war…
(Though not sure how fool-proof these are.)
And not all languages are Turing complete in the first place. Not even all useful languages.
Bash is great at dealing with files, text, running other programs, job handling, parallelism and IPC.
Those things in combination can end up being more complex in Python, which creates more opportunities for obfuscation.
And type safe.
autoconf creates a shell script by preprocessing with m4. So you need to know not just the intricacies of shell scripting, but also of m4, with its arcane rules for escaping: https://mbreen.com/m4.html#quotes
If autoconf used m4 to generate python scripts, they would also look like https://pyobfusc.com/#winners
That's AI phase one
Hey, I phase 2 is even better disguised exploit code hiding behind acres of seemingly plausible AI generated code
Also the basic workflows for the alternative build systems have maybe ten more characters to type. It's not bad.
And how did we get to the point that a complex system is required to build a compression library -- something that doesn't really have to do much more than math and memory allocation?
The project in question contained a compression library, but was not limited to it; it also contained a set of command line tools (the "xz" command and several others).
And a modern compression library needs more than just "math and memory allocation"; it also needs threads (to make use of all the available cores), which is historically not portable. You need to detect whether threads are available, and which threading library should be used (pthreads is not always the available option). And not only that, a modern compression library often needs hand-optimized assembly code, with several variants depending on the exact CPU type, the correct one possibly being known only at runtime (and it was exactly in the code to select the correct variant for the current CPU that this backdoor was hidden).
And that's before considering that this is a library. Building a dynamic library is something which has a lot of variation between operating systems. You have Windows with its DLLs, MacOS with its frameworks, modern Linux with its ELF stuff, and historically it was even worse (like old a.out-based Linux with its manually pre-allocated base address for every dynamic library in the whole system).
So yeah, if you restrict yourself to modern Linux and perhaps a couple of the BSDs, and require the correct CPU type to be selected at compilation time, you could get away with just a couple of pages of simple Makefile declarations. But once you start porting to a more diverse set of systems, you'll see it get more and more complicated. Add cross-compilation to the mix (a non-trivial amount of autotools complexity is there to make cross-compilation work well) and it gets even more complicated.
>No one has any business saying they know what something does until they've actually read it.
Beneath the placid surface of abstraction is the den of the devil.
The designer of Meson explicitly avoided making the language turing complete so for example you cannot define functions. In my experience this was an excellent decision to limit people tendency to write complex stuff and put the pressure on the Meson developer to implement themselves all the useful functionalities.
In my experience the Meson configuration are as simple as they can be and accommodate only a modicum of complexity to describe OS specific options or advanced compiler option one may need.
Please note that some projects' Meson file have been made complex because of the goal to match whatever the configure script was doing. I had in mind the crazy habits of autotools to check if the system has any possibly used function because some system may not have it.
Also, meson's build dependencies (muon, python) are a lot for some of these projects.
Same reason why I don't like TypeScript in its current form. It's not worth the extra complexity it brings.
I don't know why.
I haven’t used it for some time but autoconf always seemed like a horrible hack that was impossible to debug if it didn’t work properly.
That was bad enough back in the days where one was mostly concerned with accidents, but in more modern times things that are impossible to debug are such tempting targets for mischief.
The last build system I worked on separated build and test as separate stages. That meant you got a lot of useless artifacts pushed to a development namespace on the distribution server, but it also meant later stages only needed read access to that server.