Bash Infinity: Standard library and boilerplate framework for Bash
github.com
github.com
However, at the same time I kind of wonder what the use case would be for this. With all the extra syntax it would basically be equivalent to learning a new programming language, at which point it just seems to make more sense to use languages that have these features baked in.
While it looks like a really cool project, I could never see myself using it, as usually when I want to use a shell scripting language I want full portability, and end up writing in sh. When I want to write something more advanced I use real programming language.
You find a full version of Bash only on GNU/Linux (not even in other UNIX OS like BSD), and there you can install the interpreter for the programming language of your choice
https://google.github.io/styleguide/shell.xml "If you are writing a script that is more than 100 lines long, you should probably be writing it in something else"
I have build scripts of complex systems that are >1000 lines. They are easy to understand and maintain and I cannot image doing it in another language.
Bash is good for switching yard code, where you're gluing a lot of program invocations together with a few variables and a little bit of logic. It can certainly do other things, but that kind of code is the least risky when it becomes big.
If folks are being honest with themselves, they will find it's significantly more sane to rely on a Python interpreter being available than writing Bash scripts. Go binaries have less dependencies than any Bash script, if runtime dependencies are an issue.
There are actually quite a few incompatibilities between different implementations and versions of many of those utilities. Even if you try your best to restrict to the POSIX standard it's quite easy to "accidentally" use an extension, leading to breakage when someone runs it on Linux distro $foo or macOS.
Dependencies is a big problem in shell scripts, much more so than most other environments.
Edit: typos.
It comes with logging and unit testing. What's not to like.
There are a lot of people who does not know any other languages, believe it or not.
This is the only reason I can see to use this over perl (which it seems to be heavily influenced by) or python. Then again, learning how to use these higher level language features would probably require equivalent effort to just learning a beefier scripting lang such as those.
Also: As far as hack value, what a great project. I don't mean to bikeshed.
Still, one has to worry about and take care of programs that they depend on in their bash scripts and make sure that the correct version is installed and so, that is hard enough, I don't think adding new syntax and package management is going to help Bash, only make it more complicated and error-prone.
Bash is a programming language you don't want to use, but end up using anyway. Any effort to make it a bit less clunky and maybe even a bit saver is welcome.
*(om[1])
$i:l
(These days, I hope I can just develop my super fancy Emacs config in one place, and use TRAMP to work with remote machines through an environment customized for me.)
Wait, you use keys? Wow, man. I prefer good old-fashioned passwords, the same one on all servers, of course. And keep it memorable.
In Windows land it's quite frequent that the dev never leaves Visual Studio. In Eclipse/Java land something similar, even though often the deployments would be on Linux, for cost saving reasons, the devs would all be on Windows.
I should write a blog post or something at some point about the differences in dev careers. Enterprise software development is very different from embedded software development which is different from web development for the mass market which is different from mobile development, etc.
Don't get me wrong, those Java developers can be great guys, but when the GUI-only people have to interact with the CLI for the first time, all sorts of problems can manifest themselves.
- you've got 1 server you visit rarely so don't care about the server shell
- you've got <10 servers you care about and try to replicate your shell config (because you use it daily)
- you've got >10 servers and you almost never log into them because the logs, metrics and deployment are exported to a centralised system... so you don't care about the server shell
If you have a central home directory that is mounted on every server via nfs and zsh is installed everywhere or if you have setup some (semi-)automated file syncing/deploying and develop just on one machine that is setup correctly, then why not?
The main reason for programming Shell/Bash for me is, because its available on the tiniest of Systems, like initramfs, or routers. There aren't many options for scripting. But I also don't know if Bash Infinity makes sense there. So while this project looks interesting, I don't see me using it anytime soon.
Bash is intentionally scoped to be small. The language seems almost intentionally difficult to grok for large projects. There are very few legitimate use cases for large bash scripts, in my opinion, and frameworks only make that harder to see. Javascript also started scoped small, but it was everywhere, and look at it now...
Maybe the meaning of "large" depends on familiarity, but somehow, a big part of UNIX/Linux is a pile of large bash scripts...
Frankly, I'm amazed Bash can do any of this. I'm happy to use a fullscreen terminal all day, and come up with too-clever unix pipelines, but damn do I hate writing anything in Bash, especially when it comes to control flow.
If a program is simple enough (or the programmer determined enough) to write it in Bash, POSIX shell is a better option, as it's well defined, and well documented what does what, and how.
Relying on Bash specifically makes things much more complicated for not much benefit.
There are lots of things people "should" do, and lots of reasons why they don't; sometimes, they're good ones. So how's this for a "should"?
Prescriptive advisors should be very cognizant of the dangers caused by people following such advice without understanding why they are doing so. (Hint: the people who need such advice usually don't understand it.) Additionally, such advisors should accept responsibility when those they give advice to do really weird things while trying to follow it.
And you never know, you may someday find out that you want to move to an OS which doesn't support bash, or you start to distribute your software and the complaints roll in from the BSD users trying to port it to their OS, or bash makes some backwards-incompatible changes to some arcane behaviors this tool relies on, and since bash isn't standardized you didn't know until it was too late.
I wrote a blog post about this, if you want to read more: https://drewdevault.com/2018/02/05/Introduction-to-POSIX-she...
Just freakin learn perl or python! Do not let this beast grow any larger.
For calculations? Sure; but that's not really what bash is for.
Shells are excellent at invoking and managing subprocesses and piping. Python (not sure about Perl, I've not really used it) is terrible at those things.
Take a simple, very common piece of bash code like `foo | bar`. How might we do this in Python? Maybe we reach for the builtin `subprocess` module:
import subprocess
foo_output = subprocess.check_call(['foo'])
bar = subprocess.Popen(['bar'], stdin=subprocess.PIPE)
bar.communicate(foo_output)
OK, maybe instead of `check_call` (which is blocking) we make `foo` asynchronous. Note that we can't use `foo.communicate` to get its output, since that would also block. What if we just shuttle data between the two manually, line by line (urgh)?
import subprocess
foo = subprocess.Popen(['foo'], stdout=subprocess.PIPE)
bar = subprocess.Popen(['bar'], stdin=subprocess.PIPE)
while foo.poll() is None:
bar.stdin.write(foo.stdout.readline())
bar.stdin.close()
bar.wait()
It's at this point that we move the data shuttling into a separate thread:
import subprocess
import threading
foo = subprocess.Popen(['foo'], stdout=subprocess.PIPE)
bar = subprocess.Popen(['bar'], stdin=subprocess.PIPE)
def shuttle():
while foo.poll() is not None:
bar.stdin.write(foo.stdout.readline())
foo.stdout.close()
bar.stdin.close()
thread = threading.Thread(target=shuttle)
thread.daemon = True
thread.start()
thread.join()
bar.wait()
At this point, I'd say just freakin learn bash!
Perl is great at these things.
In fact you talk shell native tongue in Perl.
use strict;
use warnings;
my @output = `your_command | your_another_command`;
foreach my $line (@output) {
chomp; #removes new line
$line = $_;
if ($line =~ /your regex goes here/) {
#if match, use $1, $2... to get the groups matched
#your buisness logic goes here
}
}Its really more like a glue language, a step below Java.
Personally I think control structures in bash are too convoluted and easy to get lost in.
Also if you are running long-lived high-output processes and you want your program to be interactive, then of course you have to do multithreading shenanigans. At that point you're going to need a proper event queue or something at the very least. The first use case I can think of (logging dmesg -w) would definitely not work using a naive approach
foo = subprocess.Popen(['foo'], stdout=subprocess.PIPE)
bar_output = subprocess.check_output(['bar'], stdin=foo.stdout)
from subprocess import run
run('foo | bar', shell=True, check=True)Shells are excellent at invoking commands and piping, as I said.
Doing things like piping with Python instead of a shell is indeed more complicated than it needs to be. That was my point ;)
You can even do Lispy things in Perl. You can do functional stuff: https://hop.perl.plover.com/
You can do unicode regexes in Perl: https://perldoc.perl.org/perlunicode.html
You can do large file processing in Perl: https://www.perlmonks.org/?node_id=344087 , https://www.perlmonks.org/?node_id=1118102
Perl's file and string manipulation capabilities has no match.
If you use awk you have to think in the line paradigm. Perl gives you not just that but a lot more. Perl regexes are by far the strongest of their kinds programming language out there(first class entities).
In fact the whole reason why Larry Wall invented Perl is at some point in time you max out what you can do with things like awk and sed.
And I stand by what I wrote about shell + AWK, especially AWK, any day of the week.
By the by, I can do unlimited number of things in a shell program, things no other programming language can do, because I can call any other program from it. Apart from assembler, shell is the second most powerful tool because of that characteristic.
These sort of things are just the beginning. People will be surprised how hard it is to parse something like a csv.
You start to begin discovering limits when you start doing things like error handling. Writing slightly complicated regexes, or if you need a little complicated code written relying on if/for more often.
More everyday use cases that come to my mind are use of things like Data::Dumper, qw, open/while<>/close paradigm code, arrays, hashmaps, grepping over large lists, unicode, heredocs, handling binary data etc.
Nothing to take away from Bash. But its really to glue together a small bunch of unix commands in progression. If you are doing anything more than a 100 line program, you are better off with Perl.
I see you still don’t get it. I wouldn’t as an experienced shell programmer construct a JSON/XML parser; I use libxslt for XML and I compiled jq for parsing JSON. It’s the UNIX way.
The Zen of this eludes you still. Think deeper.
Seems like a blessing in disguise in this case, if such zen exists.
I should use shell+awk+sed+libxslt+jq and all that, instead of using Perl + XML::Simple?
And then there is the UNIX way, with both jq and libxslt being reusable for other data without having to write a dedicated program. sed by the way is unnecessary if one has mastered AWK.
There are too many things you have not even begun to consider yet. You could start with taking those developer-convenience glasses off, and putting the system engineer glasses on.
Believe it or not I also like shell for small tasks.
AWK was super powerful but I never had much use for it (preferred sed when I absolutely needed that kind of fuckery).
I don't look forward to encountering (read trying to fix) this in production.
I love how it took the bash to much higher level. I Bash is something you'll end up using, accepted or not. It's available by default and is the best (only?) thing to connect all the dots together when working on Linux machines, either local or servers.
This framework is lovely and indeed something that will make my life much easier.
Great job, great job!
I can think of maybe half a dozen examples in the past few years where this would have been useful. Not all IT stuff is kubernetes or plain containers, so a well-defined bash-based language is something I can see certain shops embracing.
Great work!
Looking at the modules, this is not how an experienced shell programmer would write code; functional — yes; object oriented — never. Part of the reason those of us who write in the shell do so is to escape the horror of meaningless, kilometers long stack traces and unhandled exceptions, allowing us to keep our code small and fast.
And bash, couldn’t the author have picked up ksh93 and built upon a powerful, POSIX-compliant programming language? Why aren’t we as an industry striving to be better, instead of propagating de facto fashion trends?
That said...I have my doubts: https://github.com/niieani/bash-oo-framework/issues/17
To give a specific example, even with these extensions you won't be able to handle NULL bytes in your shell script. So if you ever want to store, say, a PNG image in a variable you're screwed.
This can be a real problem. I once wrote a shell script to deal with some email data, which worked brilliant right up to the point we had to deal with emails which contained attachments that aren't base64-encoded (an external provider sent it to us that way), at which point it all came crashing down. It took me ages to discover why emails were being mangled and had to rewrite the entire stuff in Python.
It was only used in our development environment and not production, so no permanent damage. But still a waste of my time.
There are many other cases where you may want to deal with binary data.
What you're (very helpfully) pointing out is that is a case where it's just a bandaid, and Bash's flaws shine through. I suspect there will be more. But other comments are just saying "Bash bad", and those are irrelevant.
Bash is nice since it's available on many systems and is a slightly better superset of sh. However Bash is a large, ugly beast. There are a million edge cases, inconsistencies, obscure options, counter-intuitive intricacies and minute differences between versions and other shells. There's already a ridiculous amount of completely unnecessary complexity to it. I wouldn't want to increase that surface any further.
In practice, I try to keep stuff minimal, transparent and close to defaults. Usually that means -x and POSIX compliancy when possible.
CI, HealthCheck, Utility scripts(especially about extract data from log) are all valid use case for this.
Even long scripts, I have written multi thousand line programs in BASH separated into different modules.
Some of this could be helpful, but the non standard syntax kind of defeats the purpose.
Nice syntax highlighted example here >>> https://invent.life/project/bash-infinity-framework
(Thanks!)
Also, on Windows 10, you can run bash directly through Windows Subsytem for Linux. It gives you a full native Ubuntu environment through a binary compatibility layer (so it runs at near native speeds, non virtualized)