JavaScript for Shell Scripting
github.com
github.com
Ah, who am I kidding.
https://github.com/google/zx/blob/5ba6b775c4c589ecf81a41dfc9...
function substitute(arg) {
if (arg instanceof ProcessOutput) {
return arg.stdout.replace(/\n$/, '')
}
return arg
}
// […]
let cmd = pieces[0], i = 0
for (; i < args.length; i++) cmd += substitute(args[i]) + pieces[i + 1]
for (++i; i < pieces.length; i++) cmd += pieces[i]
This is the real code that a Ansible uses to run a shell command correctly and is 350 lines and is still a small subset of the features of a single line of bash. https://github.com/ansible/ansible/blob/a2776443017718f6bbd8...
The Python code to do what a single mv invocation does is 120 lines https://github.com/ansible/ansible/blob/a2776443017718f6bbd8...
People always focus on the footguns that exist in Bash the language but ignore how much systems programming is abstracted away from you in the shell environment.
In Bash you can enter a Linux namespace with a single nsenter invocation. If you want to do the same in Python you have use ctypes and call libc.clone manually.
Python has shutil.move and os.rename but the Ansible example is to illustrate that there's a lot of code that needs to surround those calls to make them useful and they're not 1-1.
- it's portable accross OSes (and keep flags if the OS supports it, deal with encoding, etc)
- it ensures selinux context is saved if there is such a thing
- it has proper and rich error communication with the calling code
- it's includes documentation and comments
- it outputs json par parsing and storage
No to say "mv" is not awesome, because it is. There is much more boiler plate in python, and is why I'll often do subprocess.check_call(['mv', 'src', 'dst']) if my script is linux only.
But you are pushing it
By the way, I don't know if this is the canonical implementation, but FreeBSD mv is 481 lines of C: https://github.com/freebsd/freebsd-src/blob/master/bin/mv/mv...
In bash, `eval` is a footgun like any other. You can use it, but you just need to be aware of where your toes are when it shoots.
It's usually a "bad idea" in the sense that if you think you need to use it, you probably don't, and 90% of the time, there is an easier way to accomplish what you want to do. The next 5% of the time, using `eval` might be easier but will also create maintenance debt with its overgeneralization. And the final 5% of the time might be actual legit use cases for `eval`.
I just grepped my codebase for `eval` and I almost never use it. One example of the "overgeneralized" 5% might be when I realized I could use `eval` to set "variable variables" (i.e. the name of the variable is itself a variable, taken from a function argument). It was cool, but I ended up deleting it in favor of a more concrete solution.
Personally, if I'm hesitating to use `eval`, it's usually not for any security reasons. In general, my bash scripts only exist in dev machines and CI runners, and I don't copy them into the application containers that are exposed to a live runtime environment with untrusted users. So for CI/dev scripts, I can safely assume the code will only run in CI/dev, and therefore I can trust arbitrary user input (which I can of course still validate).
> apostrophes in people’s surnames
This is shell scripting. We have literally decades of experience with these. We know for a positive fact that simple concatenation is dangerous. The contents of files, the names of files on the file system, and all the other things that shell scripts normally encounter are perfectly sufficient to wreck your day if you are too casual about them. Should you reply with something incredulous about this claim, be prepared for dozens of people to jump in with their war stories about how a single space in a filename in an unexpected place cost their business millions of dollars because of their shell script screwup. (Obviously, they are not that consequential on average, but the outliers are pretty rough in this case.)
Shell scripting is frankly dangerous enough just with what is on your system already; actually hooking up user input to it is borderline insane. Shell scripting would have to level up quite a bit to only be something to be concerned about when "user input" was being fed into it.
Learning to do half-decent shell scripting in most shells consists about half of learning the correct way to do things like pass through arguments from a parent to a child script, because for backwards compatibility reasons, all the easy ways are the wrong ways from the 1970s. It's nice when a more modern take on shell scripting is nicer than that.
I will also say when I'm evaluating libraries for things like shell scripting, I look for things like this, and it definitely doesn't score any points when I see stuff like this.
Yes and no. I agree with the overall premise that the footguns are well documented, but at the same time, projects like this show that there are still large segments of developers who will gleefully shoot themselves in the foot because they never took the time to learn shell, or they just never had the opportunity to earn the battle scars.
At least Google has a bug bounty program.
The point is, at this stage in the evolution of internet security, we pretty much know where the bugs come from. Injection attacks are still a huge practical problem. It would be nice if new scripting languages reduced that attack surface rather than increasing it.
I’m not sure what you’re expecting if you’re taking arbitrary IO output to build up a process with arguments separated by spaces. A lot of things had to go wrong before you get to this point.
If we’re being realistic here, this library’s likely intended use is for this is smallish scripts anyways… not large pieces of software that are creating commands on the fly to from arbitrary IO
Also, spawning node from shell to spawn shell to spawn something like ls is madness. Node has fs.readdir already and there are util packs on NPM like fs-extra and friends.
If we're actually being realistic here, we know users will use this for whatever scenario, regardless of the author's intent.
And zx's `$` could make better use of tagged template literals.
Something like this, tho it isn't correct
function $(strings, ...args) {
const cmd = [];
for (const part of strings) {
cmd.push(...part.split(/\s+/));
if (args.length > 0) {
cmd.push(args.shift());
}
}
return child_process.execFile(cmd[0], cmd.slice(1));
}
https://github.com/google/zx/blob/5ba6b775c4c589ecf81a41dfc9...
I wonder if it can work with Deno, and maybe even with Deno's security features: https://deno.land/manual/getting_started/permissions
I have also used a REST API based on the Python version to implement the same thing easily in other languages, including a nifty [elisp macro](https://github.com/NightMachinary/doom.d/blob/master/night-b...) that lets you do:
(z du -h (split-string (z ls -a) "\n" t))
For some reason there's just something that rubs me the wrong way about having a bunch of requires at the top of a bash script (or cluttering my package.json with handy tools I use once in a script). But you're gonna want things like (promisified) fs, os, chalk, readline, node-fetch, etc quite a lot of the time.
Definitely wish they'd included shelljs though; almost strange not to.
I hope they add (or already have) syntax highlighting as .mjs without the file extension (just the shebang) for GitHub, VSCode, etc.
I don't get this popularity of async-await, especially in JS where I find its combination of syntax and absence of pre-run checks overly confusing and error-prone.
And this, seriously?
await $`exit 1`This would require the scripts to no longer be vanilla JavaScript, but it seems easy to extend sucrase/babel to do that transpiling.
function pipe(...functions) {
(async () => {
let accum;
for (const fn of functions)
accum = await fn(accum);
}());
}That it extends to the $ function (anything preceding a backtick in JS is a “tagged literal” but also a function call) is just API consistency because it can’t know whether you’re calling exit versus something which actually performs IO. So it always returns a Promise.
No it's not. There are all kind of *Sync functions even in nodejs and in browsers (there was sync variant of XHR), and JS engine generally doesn't care if you block in your functions or not. JS engines don't even have an event loop, that's just a construct within the app that uses the engine, like nodejs or whatever.
console.log('start')
doA()
.then(() => {
doB()
.then(() => {
doC()
.then(() => {
console.log('finish')
})
})
})
.catch((error) => {
console.log(error)
})
console.log('start')
try {
await doA()
await doB()
await doC()
console.log('finish')
} catch (error) {
console.log(error)
} console.log('start')
doA()
.then(() => doB())
.then(() => doC())
.then(() => console.log('finish'))
.catch((error) => {
console.log(error)
}) console.log('start')
doA()
.then(doB)
.then(doC)
.then(() => console.log('finish'))
.catch((error) => {
console.log(error)
})I believe your .catch can similarly be `.catch(console.log)` for the same reason
console.log('start')
try {
doA()
doB()
doC()
console.log('finish')
} catch (error) {
console.log(error)
}
I am not a JS expert and have not had many opportunities to work with it extensively, but I confess I have always struggled with this when dealing with JS; I have often wondered if the (alleged?) performance gains from executing JavaScript like this outweighs [what I have always perceived as] the significant extra verbosity and complexity required to manage simply executing things in order.
Callback-hell and Promise-hell were real issues that plagued any project of significant size.
I remember the first time I experienced "callback hell" (2016, for me, but I'm sure it was a huge problem for others before then) when I was doing some JavaScript stuff implementing Keybase's library for GPG support - I learned they'd built a whole separate JavaScript thing called IcedCoffeeScript[1] specifically to add await/defer support to get rid of those huge callback pyramids.
I personally find it a chore to work in JS compared to other languages that work the opposite way - everything is synchronous and things become a chore when you want to deal with async stuff. Again I have limited JS experience and have never enjoyed working with it (I just am not interested in front end stuff) so I'm sure it's stuff people get used to.
Additionally, async/await allows you to wait when you need to. You don’t have to await a promise — you can just call it, and then it will work in the background. You’ll just not be able to respond when it finishes work. (Because you aren’t “wait”ing for it.)
What are some other examples with easier async in other languages? I’m just confused what you’re trying to get at. Everything in JS is synchronous. You only have async when you have async. Which is a lot, because a lot of stuff in the web is async.
The poster jmull above basically wrote[1] what I was trying to say much more succinctly; this other post[2] is also more clear.
Really I was just being snarky about the idea of shell scripting where you have to type 'await' in front of every command you want to run in a three-line file, like in the examples presented for this tool.
To clarify, JavaScript could have made await the default for async functions, so that if you called an asynchronous function without putting any keyword in front of it, execution would block until the operation completed. In that design, there would be some keyword you would use when you did not want to wait.
What they did: you have to opt-out of async execution by adding the await keyword.
What they could have done: you opt-in to async execution by adding a keyword.
They just chose the wrong default (IMO). It’s not a big deal — it’s easy enough to type “await” here and there. It’s bad, though, that you don’t really see the basic flow of control from examining the code making function calls. You also need to consult the function definitions to see which ones are async.
Another option would be to have no default, and instead require that async functions be called with an explicit indication of whether they are to execute sync or async. That too heavy-handed, IMO. Fine for a statically checked language but not a dynamic one.
I don't believe this is a language specific choice, they could have done the same with C#
I'm not seeing a benefit by wrapping almost every single command in 'await $`...`'. I get why you'd want to wrap Bash in a different language, especially when handling numbers. But I'd rather use something like Python than this verbose trickery.
And as far as I can see you don't have to await every single statement, because you can do multiple statements await $`echo 1; mkdir test; exit 0`
await Promise.all([
$`sleep 1; echo 1`,
$`sleep 2; echo 2`,
$`sleep 3; echo 3`,
])
sleep 1 && echo 1 &
sleep 2 && echo 2 &
sleep 3 && echo 3 &
wait
But you must understand that I/O in JS is async, it's how the language is build! People who are advanced in async programming find this very comfortable.
(I’ve written massive bash scripts, and now strongly prefer Ruby / python / whatever is available in the system)
Things like this are why I don't love bash. Whether a script will fail if a step fails is too important to be hidden behind a single character change in a place most people ignore.
This means that promise.all probably looks something more like
const lastWeek = DateTime.now().minus({ days: 7 }).toISOString()
const pods = JSON.parse(await $`kubectl get pods -o json`)
useValidate(pods, isKubeCtlPodList)
await Promise.all(
pods.filter(pod => !protectedPods.includes(pod.name))
.filter(pod => target.test(pod.name))
.filter(pod => pod.created < lastWeek)
.map(pod => $`kubectl delete pod ${pod.name}`)
)
In shell I can wrangle jq, date, xargs and get the same result; only this is waay easier to write, gives me validation and usable error messages, and can be altered much more easily than shell.
I write complex bash scripts somewhat often. They can be quicker to write, they are great for one-offs. But if I'm coming back to something non-trivial or if its getting deployed into production I want a language like js or python (or nearly any other language. C++, OCaml, Java all help more than they hinder).
The problem with that it doesn’t meld together with most third party libs, since they will have no knowledge of your queue and will just execute immediately or out of order.
JS keeps coming up with new ways to make this less painful, but it's ridiculous every time because it's a fundamental problem with how Node presents itself. A comical sea of "await"s in front of damn near every call is the modern version of this, and is utterly typical in real JS codebases, but before it's been callback hell, or screwing around with promises all over your codebase when 90+% of the time you just wanted things to execute in order (from your perspective), and so on.
I think a pattern where there are one or two great places at the lowest level of a Node program for program flow to act async, and then a bunch of business logic where it rarely is (probably running "under" the part where async makes sense, if you take my meaning) is far more common than those where async-friendly flow is what you want for over 50% of calls. "Call this chunk of code async, but run everything in it exactly in order" is super-common, and the interface to achieve that is exactly backwards in Node.
"Any application that can be written in JavaScript, will eventually be written in JavaScript." - Jeff Atwood
source: https://blog.codinghorror.com/the-principle-of-least-power/
Also, Shell scripting is mostly about correctly and safely interacting with other CLI-based tools - which this new thing wrapping a shell (and badly at that, see other comments in this thread) won't help you get right, either.
Very bullish on JavaScript.
There is also https://github.com/shelljs/shelljs which is implemented on top of nodejs APIs.
# Example, list all dev dependencies:
cat package.json | basho --json 'Object.keys(x.devDependencies)'bashojs seem to start in the other end, add js statement-execution to shells and and yet another pipeline-language in the mix. More like AWK with js-syntax.
Looks good as a "jq" on steroids but i wouldn't compare it to shelljs.
`files`: https://github.com/franciscop/files/
import { read, walk } from 'files';
// Find all of the readmes
const readmes = await walk('demo')
.filter(/\/readme\.md$/) // Works as expected!
.map(read);
console.log(readmes);
// ['# files', '# sub-dir', ...]
import cmd from 'atocha';
// Any basic command will work
console.log(await cmd('ls'));
// Using a better Promise interface, see the lib `swear`
console.log(await cmd('ls').split('\n'));
// Can pipe commands as normal
console.log(await cmd('sort record.txt | uniq'));
// Without `swear`:
const list = await walk('demo');
const readmes = list.filter(item => /\/readme\.md$/.test(file));
const content = await Promise.all(readmes.map(read);
// With `swear`:
constn content = await walk('demo').filter(/\/readme\.md$/).map(read);This short wrapper is great for doing asynchronous SSH commands. https://github.com/gpasq/deno-exec
Coming from Deno, the single biggest advantage I see here is the handy tag function $:
await $`cat package.json | grep name`
Reading the docs [1], it appears there is a `raw` property which might make this even simpler?
[1]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
But if portability was an issue I would still probably move to something that outputs a single binary (Go, Rust).
I just think bash is not a very good language compared to the alternatives.
for x in a b c; ...
Maybe C could work, but I have to make sure it's statically linked.
Neither is Bash, despite belief to the contrary.
[1]: https://docs.oracle.com/javase/8/docs/technotes/guides/scrip...
It also supports some Nashorn features for interop with Java, but for many things to work, you need to enable it explicitly, especially the compatibility mode flag (`-Dpolyglot.js.nashorn-compat=true`).
See https://github.com/oracle/graaljs/blob/master/docs/user/Nash...
Cool story.
That being said, I have grown very fond of writing "plumbing"-scripts in nodeJS. The syntax is so much more sane than bash, the documentation of nodeJS is actually pretty good in my opinion, and once nodeJS is installed and available, its super easy to mash something together to get working. For small scripts, the uglier parts of JS do not really matter, and while nodeJS does have some APIs that feel a bit odd to use at times, it's at least well-documented with examples/guides all over the web.
Some folks use python for that exact purpose, but I always felt that getting something going takes me more effort than nodeJS (the fact that I'm just not fond of Python as a language might also have something to do with this).
All in all... for folks still using bash, I really recommend to give JS a try.
I mean nodejs =/= javascript, the problem is nodejs and some of the people who designed it who basically made it so that it would rely heavily on NPM, a commercial solution, thanks to the virtually non existing standard library. Now Microsoft virtually owns Nodejs since it owns NPM. Nodejs creator himself regretted this decision, publicly.
Sure, doing things like http directly with node is a bit cumbersome and having a library that does some abstraction comes in handy if you're working on something that goes way beyond what you'd ever consider doing in bash. But especially for that very purpose of things where I used nodeJS instead of using bash, nodeJS always had everything I needed out-of-the-box, almost never required any dependencies.
That being said, if you re-use code, plan to share scripts etc. with other people and all these things - using something like typescript (and possibly some testing-libraries) comes in very handy for maintainability, and there you can't get around npm. But for one-off-"plubming-scripts", nodeJS had all I ever needed.
It's pre-installed on a lot of systems, backwards compatibility is good, and despite the inconsistencies, the standard lib can do a lot of stuff.
https://gist.github.com/tored/b500eb7c10fbabbe2043126e51caf2...
It would be nice to have something like this more integrated within the shell or maybe composer itself.
But just for "one-off"-scripts, JS usually does the job pretty well. For bigger scripts or scripts where I want to reuse/share parts, typescript is an option - but it comes with a few hurdles: Most of the scripts I'm talking about work with zero npm-dependencies, while typescript adds at least one.
Furthermore, I either have to take care of compiling it, or use ts-node - one more dependency; and in both cases, I can't avoid the usual es6 vs. commonJS shenanigans. It's a bit of overhead to get going; always worth it if I spend more than a few hours on a script, but I can do without if it's just a few lines that I will hardly ever see again.
It isn't, that project looks like perl re-invented but worse, since it mixes Javascript via nodejs and shell script syntax.
Shell scripts already support loops, functions, conditions, variables, ... it makes more sense to write a nodejs script that works as a regular shell process and include it in a shell script than the other way around...
It's merely doctrinal orthodoxy full of needless and ceremonious frivolities for the adherents to chin stroke in approval.
It is not, in any way, a reasonable way to do things, to exercise engineering, write maintainable or reliable code or otherwise accomplish tasks.
It is just spectacle.
PHP used to be the utility knife for a while for the same reasons. It was easy to learn, and run it. And it was ok to write bad PHP code. I remember most of the exploits and remote shells were written with PHP because it was the easiest language for hackers to learn.
That would be Ruby for me, but I also accept Python and maybe dozens of others.
I generally use "spawn" from child_process instead of "exec" (which this tool uses), which can pipe output to the terminal as it happens. Great for creating build scripts.
await mycall()
mycall()!
I mean I love JS as much as the next person, but perfect? No way.
Their own readme isn’t trying to suggest that it’s an improvement, just some sort of convenience.
POSIX shells (and OS) are the common thread underlying all *nix-based software.
Bash prioritizes terseness and backwards compatibility; which is occasionally contradictory with UX, but are valid prioritizations nonetheless.
Also, learning something new gives you a more nuanced appreciation of what you already knew. It’s a mind-expanding experience, even if it’s occasionally frustrating.
TLDR. I’d advocate for it. It’s OK if things break or fall apart on the road, this is the price of knowledge acquisition.
Do you have any suggestions on how one should go about learning it?
If I do some course right now, then in a year's time if I use it maybe on few occasions in a year, I won't remember it, and it will stop being practical again.
Edit: that parallel rsync example answers my question nicely. I should have read more carefully.
Born out of frustration with bash and Python.
and ... nope, never considered JS for that type of scripting.
Awaiting citations on usually, convenient, and perfect.
> await $`cat package.json | grep name`
I mean, in the first line they're well into "I don't understand shells" territory with a superfluous cat pipe (you can just grep name package.json). This is seriously published under the google banner?
It seems like Bash has tons of footguns and unintuitive syntax. Is this just because the language grew organically?
(Of course jQuery's main job is DOM manipulation which isn't needed here)