Biggest shell programs
github.com
github.com
I jumped in and started digging around, and to my horror, the OMS was a giant set of shell scripts running on an AIX server, which evolved over a decade and was abandoned. It was over 50,000 lines of code! It was horrendous and shit kept timing out everywhere -- orders, payments, and other information were moved from server to server over FTP, parsed with complicated sed/awk, and inventory was tracked in text files (also FTPd around.)
At the time, perl seemed like the most practical way for me to migrate the mess -- I rewrote all of the shell piece by piece, starting with the simplest peices and replaced them with small perl modules as part of a larger perl application, refactoring along the way. It took me 3 months and I moved the whole thing to about 5000 lines of perl, and it ran 10-100x faster with almost none of the failures in the original system.
As terrible as it was, it's one of the most satisfying things I've ever done. :-)
That's deleting 800 lines a day, each day. Did you need to read through the original code, get a deep understanding and match its behavior exactly or did you throw away huge chunks and write new code as you thought it should behave?
Was there a lot of boilerplate that could be replaced quickly?
50,000/90 = 555 ???
OSH is the most bash-compatible shell in the world, and YSH is a new language
ls | sort | uniq | wc -l # this is both OSH and YSH
var mydict = {foo: 42, bar: ['a', 'b']} # this is new YSH stuff you can start using
json write (mydict)
I think it's worth it for 2 things alone
- YSH checks all errors - you never lose an exit code
- YSH has real arrays and doesn't mangle your variables with word splitting
There's a lot more: modules with namespaces (use mymodule.ysh), buffered I/O that's not slow, etc.
Gradually upgrading -https://github.com/oils-for-unix/oils/wiki/Gradually-Upgradi... (people are writing new YSH, but not many people have gradually upgraded, so I'd definitely appreciate feedback from people with a big "shell script problem")
---
There is a FAQ here about Perl:
Are you reinventing Perl? - https://www.oilshell.org/blog/2021/01/why-a-new-shell.html#a...
Not to say that migrating to Perl is worse in any way, i.e. if you already know Perl or your team knows it.
But objectively YSH is also a shell, so I think more of the code carries over, and there is a more direct upgrade path.
---
[1] Unix Shell Should Evolve like Perl 5 - https://www.oilshell.org/blog/2020/07/blog-roadmap.html#the-... - i.e. with compatible upgrade options
Haskell (e.g. shh) and Clojure (Babashka) are also a nice for this usecase, but more niche options.
The first really big one I wrote was the ~7000 line installer for the Enrust CA and directory, which ran on, well, all Unixes at that time. It didn't initially, of course, but it grew with customer demand.
The installation itself wasn't especially complicated, but upgrades were, a little, and this was back when every utility on every Unix had slight variations.
Much of the script was figuring out and managing those differences, much was error detection and recovery and rollback, some was a very primitive form of package and dependency management....
DEC's Unix (the other one, not Ultrix) was the most baffling. It took me days to realize that all command line utilities truncated their output at column width. Every single one. Over 30 years later and that one still stands out.
Every release of HP-UX had breaking changes, and we covered 6.5 to 11, IIRC. I barely remember Ultrix or the Novell one or Next, or Sequent. I do remember AIX as being weird but I don't remember why. And of course even Sun's three/four OS's had their differences (SunOS pre 4.1.3; 4.1.3; Solaris pre 2; and 2+) but they had great FMs. The best.
Weirdly, today I ran wish in MacOS Sequoia (15.1.x) and had the (exception) output truncated at terminal width!
What you say here contains some truth (certainly with respect to the kernel), but I doubt that has anything to do with the behaviour the grandparent is reporting–the wish command in Tcl/Tk truncating output at terminal width. That behaviour would be determined by the Tcl/Tk code, nothing inherently to do with the underlying OS.
> but OSF/1 which, under few different names, was sold by Digital as Unix for Alpha (there were few rare builds for MIPS too).
IBM also briefly sold a port of OSF/1 to IBM mainframes, AIX/ESA: it was available to customers in June 1992, and withdrawn from marketing in June 1993–it was discontinued so quickly due to lack of customer interest, and also because IBM was about to release (in 1994) a UNIX compatibility subsystem for MVS (OpenEdition), which was a more attractive UNIX option for many of their mainframe customers.
I believe IBM's abortive Workplace OS – shipped in beta form only as OS/2 PowerPC Edition – was also partially derived from the OSF/1 code base.
But I just tried it again on a resized terminal window and I couldn't reproduce it!
Also, several APIs introduced after BSD 4.4 are very visibly missing, pointing to how little was taken from Free/NetBSD
As suggested above, it did this even when called from a script.
The fix was easy, set COLUMNS ridiculously large if DEC Unix, but it took days of WTF apparent UB before I realized how simple was what was happening. It just seemed haphazard: I'd reposition and resize a window so I could run the script in one while manually running the commands in another, get inconsistent results, rinse, repeat...
...and eventually realize the common element in each test was me, and the variations I was introducing were window size.
I cursed their engineers for trying to be "helpful" and keep things "pretty".
Do you mean OSF1/, Digital Unix, or Tru64 Unix?
Technically OSF/1 was supposed to be the commercial BSD answer to System V, in practice only several niche vendors used it, plus Digital and NeXT (and through NeXT, Apple which continues the line to this day)
Other than the COLUMNS thing. That is burnt into my memory forever.
It's 6224 lines, so far.
There is a top-level binary with sub-functions, sort of like how
git [ git options ] < git action> [action options]
systemctl [etc.[
There is a sub command to add a new sub command, which creates the necessary libraries and pre-populates function definitions from a template; the template includes short and long usage functions, so that
cbap -h
cbap pipeline -h
There are subcommands for manipulating base images, components (which are images with specific properties for use as containers in the pipelines), and pipelines themselves. A LOT of code is for testing, to make sure that the component and pipeline definitions are correctly formatted. (Pipelines are specified in something-almost-TOML, so there is code to parse toml, convert sections to arrays, etc., while components are specified as simple key=value files, so there is code to parse those, extract LHS and RHS, perform schema validation, etc.).
Since pipeline components can share properties, there is code to find common properties in var and etc files, specify component properties, etc.
There are a lot user and group and directory and FIFO manipulation functions tailored to the security requirements: When a pipeline is setup, users and groups and SEL types and MCS categories are generated and applied, then mapped into to the service files that start the components (so there is a lot of systemd manipulation as well).
Probably the single biggest set of calls are the functions that get/set component properties (which are really container properties) and allow us to use data-driven container definitions, with each property having a get function, a validation function, and an inline (in a pipeline) version, for maximum flexibility.
Finally, there is code that uses a lot of bash references to set variables either from files, the environment, or the command line, so that we can test rapidly.
It also support four levels of user, from maintainer (people who work on the code itself), developer (people who develop component definitions), integrators (people who build pipelines from components), and operators (people who install pipelines), with the ability to copy and package itself for export to users at any of those levels (there is a lot of data-driven, limited recursive stuff happening therein).
Since target systems can be any Linux, it uses makeself to package and extract itself.
For example, an integrator can create a pipeline definition, which will produce a makeself file that, when run on the target system, will create all users, groups, directories, FIFOs (the inter-component IPC), apply DAC and MAC, create systemd files, copy images to each user, and launch the pipeline - with a delete option to undo all of that.
There is some seccomp in there as well, but we've paused that as we need to find the right balance between allow- and deny- listing.
(Yes, I use shellcheck. Religiously. :->)
Just discovered this myself, also trying to make a language target shell. Was really surprised bc/dc was not present I think in Ubuntu install in WSL2. Also using awk for floating point math, but just shelling out to it.
https://pubs.opengroup.org/onlinepubs/9699919799.2008edition...
When people do large quantity text and OS interaction work, languages like Java and Python are a giant pain. And you will begin to notice how Shell/Perl become a breeze to do this kind of work.
This means nearly every automation task, chaotic non-standard interfaces, working with text/log files, or other data formats that are not structured(or at least well enough). Add to this Perl's commitment towards backwards compatibility, a large install base and performance. You have 0 alternatives apart from Perl if you are working to these kind of tasks.
I have long believed that a big reason for so much manual drudgery these days, with large companies hiring thousands of people to do trivially easy to automate tasks is because Perl usage dropped. People attempt to use Python or Java to do some big automation tasks and quit soon enough when they are faced with the magnitude of verbosity and overall size of code they have to churn and maintain to get it done.
Also, if end user is on Windows, there is already Perl like option on their desktop, it's called Powershell and will perform similar to Perl.
As a bonus you can use single code base for everything no matter if there's http or something else in the line.
A few links for context:
Are you reinventing Perl?
https://www.oilshell.org/blog/2021/01/why-a-new-shell.html#a...
The Unix Shell Should Evolve Like Perl 5 (with compatible upgrade options, rather than a big bang like Perl 6/Raku)
https://www.oilshell.org/blog/2020/07/blog-roadmap.html#the-...
A Tour of YSH - https://www.oilshell.org/release/latest/doc/ysh-tour.html
That may not seem like a big advantage until you're working in an environment where you don't actually have the advantage of just installing things from the open Internet (or reaching the Internet at all).
First, there’s the issue of readability. Bash's syntax can become downright cryptic as it grows. Variable scoping rules are subtle, error handling is primitive, and string handling quickly becomes messy. These factors translate into code that’s harder to maintain and reason about. As a result, future maintainers are likely to waste time deciphering what’s going on, and they’ll also have a harder time confidently making changes.
Next, there’s the lack of robust tooling. With more mature languages, you get static analysis tools, linters, and debuggers that help you spot common mistakes early on. For bash, most of these are either missing or extremely limited. Without these guardrails, large bash programs are more prone to silent errors, regressions, and subtle bugs.
Then there’s testing. While you can test bash scripts, the process is often more cumbersome. Complex logic or data structures make it even trickier. Plus, handling edge cases—like whitespace in filenames or unexpected environment conditions—means you end up writing a ton of defensive code that’s painful to verify thoroughly.
Finally, the ecosystem just isn’t built for large-scale Bash development. You lose out on modularity, package management, standardized dependency handling, and all the other modern development patterns that languages like Python or Go provide. Over time, these deficits accumulate and slow you down.
I think using Bash for one-off tasks or simple automation is fine — it's what it’s good at. But when you start thinking of building something substantial, you’re usually better off reaching for a language designed for building and maintaining complex applications. It saves time in the long run, even if the initial learning curve or setup might be slightly higher.
I don't think c99 would be a good choice as processors will likely be different in 30 years time. If you had your program on e.g. a usb stick and you manage to load it onto a machine, it'd only be able to run if you had the same architecture. Even nowadays, you'd run into difficulties with arm and x86 differences.
Some kind of bytecode language might seem better (e.g. java), but I have my doubts about backwards compatibility. I wonder if Java code from 20 years ago would just run happily on a new Java version. However, there's also the issue of Java not being installed everywhere.
Absolutely.
Personally, I've encountered great difficulties with some old SAN software that required a Java 6 web plugin that I couldn't get running on anything other than Internet Explorer - I kept an XP VM with the correct version just for that. I suspect a large part of the problem was the software incorrectly attempts to check that the version is at least 6, but fails when the version is newer (they obviously didn't test it when later versions got released).
The script works but it always feels like something is going to break if I look at the code the wrong way.
See my comment here, with some details: https://news.ycombinator.com/item?id=42354095
(I created the project and the wiki page. Right now the best bet is to join https://oilshell.zulipchat.com/ if it interests you. People who want to test it out should be comfortable with compiling source tarballs, which is generally trivial because shells have almost no dependencies.)
The first step is:
shopt --set strict:all # at the top of the file
shopt -s strict:all 2>/dev/null || true
OSH should run your script exactly the same as bash, but with better error messages, and precise source locations.
And you will get some strictness errors, which can help catch coding bugs. It's a little like ShellCheck, except it can detect things at runtime, whereas ShellCheck can't.
[2] https://git.einval.com/cgi-bin/gitweb.cgi?p=abcde.git;a=blob...
Allegedly.
Note that much of the code size of these scripts is dedicated to ensuring that the right utilities exist across the various platforms and perform as expected with their various command line options. This is the worst pain point of any serious shell script author, even worse than signals and subprocesses (unless one enjoys the pain).
*Information that, I would argue, would be less transparent if rkhunter had been written in a "proper" programming language. It might be shoved off in some records in data structures to be retrieved; actions might be complex combinations of various functions---or, woe, methods and classes---on nested data structures; logging could be JSON-Bourned into pieces and compressed in some database to be accessed via other methods and so on.
Shell scripts, precisely due to the lack of such complex tools, tend to "spill the beans" on what is happening. This makes rkhunter, for instance, a decent documentation of various exploits and rootkits without having to dig into file upon file, structure upon structure, DB upon DB.
The code which builds the updates probably adds up to more lines, but that's split across many files.
https://github.com/acmesh-official/acme.sh/blob/master/acme....
That and the CA that was exploiting shell injection in acme.sh convinced me it was time to move on
Targeting busybox may be your best bet, but once you are leaving your typical Linux system, writing portable (Bourne) shell scripts becomes hard to impossible.
Historically people used to sell compilers - so minimizing installation to dev machines probably was a savings (and in those times space was at a premium everywhere).
That said - I am with you, give me any other programming language besides shell!
But people have told me stories about working for the government
(my background was more "big tech", and video games, which are both extremely different)
Some government/defense systems are extremely locked down, and they don't have C compilers
So people make do with crazy shell script hacks. This is obviously suboptimal, but it is not that surprising in retrospect!
FWIW, there is an apple.com .dmg "command line tools" that is much smaller than Xcode formal, but I actually came here to say that to the very best of my knowledge a $(ruby -e $(curl ...)) to brew install will then subsequently download pre-built binaries from GitHub's docker registry
I count 10k lines of hand-written bash in the system tests:
$ git clone git@github.com:apache/incubator-pagespeed-mod.git
$ git clone git@github.com:apache/incubator-pagespeed-ngx.git
$ find incubator-pagespeed-* | \
grep sh$ | \
grep system_test | \
xargs cat | \
wc -l
10579(I'm very conflicted about learning new things that make shell more convenient: I don't need more things pushing me toward using this in-many-ways-horrible-but-I'm-so-fast-with-it tool.)
My old finger memory will still probably put them in, alas.
hah, found it. Byte, 1998: https://archive.org/details/199806_byte_magazine_vol_23_06_w...
I tried metacard after reading that. It ran on linux: http://www.sai.msu.su/sal/F/5/metacard.gif ... I think I might have written some things with it. Cool, good luck on me trying to find 26 year old software.
You can totally still run this if you want btw - just download some old linux ISOs from archive.org and install it in a VM ... hope they survive all their lawsuits; we're so lucky to have them around.
If it was up, it would do a speedcheck and record that for the IP the VPN was using, then check to see how that speed was compared to the average, with a standard deviation and z-score. It would then calculate how long it should wait before it recycled the VPN client. Slow VPN endpoints would cycle quicker, faster ones would wait longer to cycle. Speeds outsize a standard deviation or so would check quicker than the last delta, within 1 Z would expand the delta before it checked again.
Another one about that size would, based on current time, scrape the local weather and sunup/sundown times for my lat/long, and determine how long to wait before turning on an outdoor hose, and for how long to run it via X10 with a switch on the laptop that was using a serial port to hook into the X10 devices. The hose was attached to a sprinkler on my roof which would spray down the roof to cool it off. Hotter (and sunnier) weather would run longer and wait shorter, and vice versa. I live in the US South where shedding those BTUs via evaporation did make a difference in my air conditioning power use.
Having said that, if I were to start experimenting with an altogether different shell, I would be very tempted to try jshell!
Incidentally, I hate when projects say stuff like "Oils is our upgrade path from bash to a better language and runtime". Whether a change of this kind is an "upgrade" is completely subjective, and the wording is unnecessarily haughty / dismissive. And very often you realise that projects who say that kind of thing are basically just using the underlying tech wrongly, and trying to reinvent the wheel.
Honestly, I've almost developed a knee reflex to seeing the words "upgrade" and "better" in this kind of context by now. Oils may be a cool project but that description is not making me want to find out more about it.
I meant that, if I come up with a project called "Spills: an improved Oils without all the awful warts", this says nothing good or meaningful about my project itself, and all it really does is make a casual implication that Oils is crap. So such a sentence would (personally) put me off Spills, rather than get me excited about it. Especially if I was already happy with Oils and in fact found it enjoyable, and certainly not having "awful warts".
Yes you can go 'delve' into the project to figure out if and why Spills is actually better than Oils (and if and why Oils is 'crap' according to your tagline), but a tagline is chosen for a reason. It's a single sentence you use to describe and sell your project with. If your best tagline is "that other product is crap" then I'm not that interested to do any 'delving' in the first place. That was my point.
Also, I think partly the reason bash gets a bad rep is because it's a scripting language, and the canonical one for that matter. A lot of the time people get frustrated with bash, I find it's because they're trying to use it as a 'system' language and get everything done via bash. But you're not supposed to. It's a scripting language, it's intended as 'glue'. Where system languages rely on external libraries, bash relies on external programs. You want to validate your inputs? Use a validator program. You want to operate on specific types? Use a type-specific program. If you need type-specific validation and you're doing it in bash, then of course you're going to get frustrated; it probably can be done, and possibly even well, but the language was just never designed for that kind of thing and you'd have to get knee-deep into arcane hackery rather than have lovely clean maintainable code.
$ file /usr/bin/* | grep "shell script" | cut -f1 -d':' | xargs wc -l | sort -n
6431 /usr/bin/tkcon
1030 /usr/bin/dtruss
I'm an SRE for a service everyone has heard of. I have inadvertently pasted into my terminal prompt multiple times now, which has attempted to run each line as a command. I see there is a way to disable this at the shell for each client, but what about at the server level? This way I could enforce it as a policy, and not have to protect every single user (including myself) individually. Said differently, I want to keep everyone who ssh into a prod machine from being able to paste and execute multiple lines. But not forbid paste entirely.
The only thing I could think of would be to recompile bash and detect if the input was from a tty. If so, require at least 200ms between commands, and error out if the threshold exceeded. This would still allow the first pasted command to run, however.
It inserts a control sequence before and after the pasted contents that makes bash not execute all the lines. Instead it will keep them in the command line, post which you can choose to execute all of them in one go with enter or cancel with ctrl C.
bind 'set enable-bracketed-paste on'Imagine you want to run, say, Firefox like that (say because you'd like to see stdin/stderr output of what's going on without having to find to which log file it's outputting stuff: it's really just a silly example):
xterm> firefox
<-- the xterm is now "stuck here" (until Firefox exists)
Hello, world!
xterm> firefox
<-- the xterm is now "stuck" here (until Firefox exits)
cat > /dev/null
So, yup, a good old cat > /dev/null works wonder.
so whatever you do, it should be a feature, even defaulted on. but never a policy that you enforce to "everyone who ssh into a prod machine"
if you find something that works well for you, add it as a suggestion to your developer docs.
27k lines/24k loc
It began:
# Yeah yeah I know
https://relax-and-recover.org/
This is the equivalent of the "Ignite" tool under HP-UX.
Looks legit to me, e.g. https://github.com/rear/rear/blob/master/usr/share/rear/lib/...
I tried opening it up just to look at it and most text editors just absolutely choked on it. I can't remember, but it was either Vim xor Emacs that could finally handle opening it.