Scripting with Go (2022)
bitfieldconsulting.com
bitfieldconsulting.com
# the original one-liner to get unique IP addresses
cut -d' ' -f 1 access.log | sort | uniq -c | sort -rn | head
# turns into this with GNU awk
gawk '{PROCINFO["sorted_in"] = "@val_num_desc"; a[$1]++} END {c=0; for (i in a) if (c++ < 10) print a[i], i}' access.log
$ wc -lc fake_log.txt
1000000 218433264 fake_log.txt
$ hyperfine "gawk '{PROCINFO[\"sorted_in\"] = \"@val_num_desc\"; a[\$1]++} END {c=0; for (i in a) if (c++ <10) print a[i], i}' fake_log.txt"
Benchmark 1: gawk '{PROCINFO["sorted_in"] = "@val_num_desc"; a[$1]++} END {c=0; for (i in a) if (c++ <10) print a[i], i}' fake_log.txt
Time (mean ± σ): 1.250 s ± 0.003 s [User: 1.185 s, System: 0.061 s]
Range (min … max): 1.246 s … 1.254 s 10 runs
$ hyperfine "cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head"
Benchmark 1: cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head
Time (mean ± σ): 4.844 s ± 0.020 s [User: 5.367 s, System: 0.087 s]
Range (min … max): 4.817 s … 4.873 s 10 runs
$ hyperfine "gcut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head"
Benchmark 1: gcut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head
Time (mean ± σ): 3.622 s ± 0.004 s [User: 4.149 s, System: 0.078 s]
Range (min … max): 3.616 s … 3.629 s 10 runs Original: 3.294s
w/ LC_ALL=C: 1.055s
w/ larger sort buffer `-S5%`: 0.780s
Your gawk: 1.772s
+ LC_ALL=C: 1.772s
I did find one other interesting difference between BSD and GNU tools - BSD sort defaults to 90% for its buffer, GNU sort defaults to 1024 KiB.
Combining all of these (and using GNU uniq - it was also faster), I was able to get down to 463 msec on the M1 Air:
$ hyperfine "export LC_ALL=C; gcut -d' ' -f1 fake_log.txt | gsort -S5% | guniq -c | gsort -rn -S5% | head"
Benchmark 1: export LC_ALL=C; gcut -d' ' -f1 fake_log.txt | gsort -S5% | guniq -c | gsort -rn -S5% | head
Time (mean ± σ): 463.4 ms ± 3.3 ms [User: 965.5 ms, System: 93.3 ms]
Range (min … max): 459.9 ms … 469.8 ms 10 runs
$ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head"
Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head
Time (mean ± σ): 1.504 s ± 0.318 s [User: 2.833 s, System: 0.474 s]
Range (min … max): 0.942 s … 1.937 s 10 runs
$ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head"
Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head
Time (mean ± σ): 3.847 s ± 0.093 s [User: 4.165 s, System: 0.613 s]
Range (min … max): 3.591 s … 3.919 s 10 runs
Syscalls aren't free, nor are disk reads, so if you have the RAM to support slurping the entire file at once, it's sometimes faster.
I can sympathize with the author w.r.t wanting to use a single language you like for everything. However, after decades I've found this to be untenable. There are languages that are just simply better for one-off scripting (Perl, Python), and languages that aren't (anything compiled). Trying to bolt an interpreter onto a compiled language from the outside seems like a lot of work for questionable gain.
One reason is deployment. Writing code in python/node/etc... implies the ability of the production environment to bootstrap a rather complicated installation tree for the elaborate runtimes required by the code and all its dependencies. And so there are elaborate tools (npm, venv, Docker, etc...) that have grown up around those requirements.
Compiled languages (and Go in particular shines here) spit out a near-dependency-free[1] binary you can drop on the target without fuss.
I deal with this in my day job pretty routinely. Chromebooks have an old python and limited ability to pull down dependencies for quick test runs. Static test binaries make things a lot easier.
[1] Though there are shared libraries and runtime frameworks there too. You can't deploy a Gnome 3 app with the same freedom you can a TCP daemon, obviously.
I think it's more accurate to say "static binaries" instead of "compiled languages." The same headaches exist with dynamically linked, compiled binaries (and sometimes they're worse since you don't have a dependency manager, unless you add one)
> Static test binaries make things a lot easier I think this really depends on your target environment and how much control you have over it. If you're in a Ruby or Python shop, for example, all your servers already have the stack installed. If you're targeting end user devices, those can have a huge mess of different configs to account for
It's nothing like that with Python or Node. The rule there is that you get something working locally and then spend a while reverse engineering a pip/venv recipe or manifest or whatever to make it work somewhere else. It's decidedly non-trivial.
But I think for python you could also deploy a binary with pyinstaller.
I could imagine a lifetime of collecting scripting macros/libs in lisp to be as good or better.
However, the reason Bash is so prolific amongst Sys Admins such as myself is the fact that they are portable and reliable to use across Debian, Arch or RHEL based distributions.
You don't have to import extra libraries, ensure that you are running the proper python environment, or be certain that pip is properly installed and configured for whatever extra source code beyond what is included out of the box.
Bash is the most consistent code you can write to perform any task you need when you have to work with Linux.
Python is (at least in the CPython implementation) compiled, to python byte code which runs on the python virtual machine.
Its not compiled to native code. (Unless you use one of the compilers which do compile it to native code, though they tend to support only a subset of python.)
Once you use it to manage complex data structures and flow, you are simply wasting time because you will have to rewrite it in Python or Go.
Just like your post does! :)
Until you try to use a newer feature or try the script in a Mac or BSD or any older bash.
SH code is completely portable, but bash itself can have quite a few novel features. Don’t get me wrong - I’m happy the language is dynamic and still growing. But it can make things awkward when trying to use a script from a newer system on an older server (and the author has been “clever”).
Not exactly.
https://stackoverflow.com/questions/11376975/is-there-a-mini...
Installing python or nodejs into your distro will inflate it by at least 30mb, quite crucial when dealing with containers
Not false, but perhaps in need of better definition. The term script has often denoted a trivial set of commands run by $interpreter.
"Scripting languages" have been seen as being in contrast to C, C++, Pascal, Java, SmallTalk, &c. The scripting languages remove from the user the need:
-a- to think about an extensive type system,
-b- to compile the logic, and
-c- to build for a specific architecture.
That requires a level of bookkeeping which is helpful for large programs and a nuicense for small programs.
I do have to add that Python more than any other language I've used results in working the first try that it's not surprising.
actually amount of reasoning, which program requires to perform in run time is close to interpretion.
Yes, and if the thing I'm trying to do has a small input, it will only be done once, etc. I will often just pipe `grep` to `sort` or whatever, because it's less typing, it's generally clearer to a wider range of people, etc.
But on larger inputs, or even things like doing a single pattern inversion mixed with a pattern match, I like awk.
I've made this point before, but I still find it hilarious. For more than a decade, awk was dead. Like, dead dead. There was nothing you could do in awk that wasn't cleaner and simpler and vastly more extensible in perl. And, yes, perl was faster than gawk, just like gawk is faster than shell pipelines.
Then python got big, people decided that they didn't want to use perl for big projects[1], and so perl went out of vogue and got dropped even for the stuff it did (and continues to do) really well. Then a new generation came along having never learned perl, and...
... have apparently rediscovered awk?
[1] Also the perl 5 tree stagnated[2] as all the stakeholders wandered off into the weeds to think about some new language. They're all still out there, AFAIK.
[2] Around 2000-2005, perl was The Language to be seen writing your new stuff in, so e.g. bioinformatics landed there and not elsewhere. But by 2015, the TensorFlow people wouldn't be caught dead writing perl.
Also the classic sysadmin role which used to lean on Perl heavily sort of evolved with rise of The Cloud and automation tools like Chef, Puppet, and Ansible took over in that 2005-2015 time frame.
By the way, I think this is why Perl lost to Python on larger scripting and programming projects - it's just easier to write (albeit harder to read, to antagoinze the Python lovers out there).
AWK, on the other hand, is simplicity itself. Sure, it misses a few things, but for searching through log files or db dumps it's an excellent tool. And it's fast enough. If you really need much more speed, there are other tools, but I would rather rewrite it in C than try perl again.
And in a world where perl is a natural choice for these tasks, awk doesn't have a niche. Because at the end of the day awk is simply an inferior language.
Which is the bit I find funny: we threw out and forgot about a great tool, and now we think that the ancestral toy it replaced is a good idea again.
EDIT: I decided to ask GPT-4 to translate the gawk script to Perl. I make zero claims that this is ideal (as stated, I don't know Perl at all), but it _does_ produce the same output, but slightly slower than the gawk script.
$ hyperfine "perl -lane '\$ips{\$F[0]}++; END {print \"\$ips{\$_} \$_\" for (sort {\$ips{\$b} <=> \$ips{\$a}} keys %ips)[0..9]}' fake_log.txt"
Benchmark 1: perl -lane '$ips{$F[0]}++; END {print "$ips{$_} $_" for (sort {$ips{$b} <=> $ips{$a}} keys %ips)[0..9]}' fake_log.txt
Time (mean ± σ): 1.499 s ± 0.006 s [User: 1.447 s, System: 0.050 s]
Range (min … max): 1.490 s … 1.507 s 10 runs perl -e 'for $i (<>) { $i =~ s/ .*//; push @list, $i; }; print(sort(@list));'Learning awk a bit more deeply in recent years has been good too. I can write one liners that do more. I shipped a full awk script once, for something unimportant, but I would never do that again. For serious text munging these days I'd rather write a Rust program.
The script package looks really cool and I'll definitely try it out, cause honestly even though I do a lot of bash scripting it's super painful for anything but something super simple.
{
PROCINFO["sorted_in"] = "@val_num_desc"
top_ips[$1]++
}
END {
counter = 0
for (i in top_ips) {
if (counter++ < 10) {
print top_ips[i], i
}
}
}
At least half of what people complain about with shell scripts can be solved by using ShellCheck [0], and understanding what it's asking you to do. I disagree with the common opinion of "anything beyond a few lines should be a Python script instead." If you're careful with variable scoping and error handling, bash is perfectly functional for many uses.
I don't think this is true. Before I learned Go, I could follow along most Go programs pretty well, and learning Go well enough to get started took less than an hour. Every attempt I've made to learn more Awk, I've bounced off.
Whereas if you know awk and bash, then they can be incredibly useful in a pinch. It doesn’t knock how powerful they are. I think it is worth learning. But if something needs to be maintained then there is an argument for Python/Go/whatever.
“Loaded guns are perfectly functional for juggling, just be careful with the trigger and you won’t shoot yourself in the foot!”
You are technically correct but why bother with being careful when you could just avoid writing bash?
func Must[T any](x T, err error) T {
if err != nil {
panic(err)
}
return x
}
In the event that you want to handle errors in some other manner, you trivially can, and you're not limited to just the pipeline design patterns, which are cool in some ways, but limiting when that's all you have. (It can also be tricky to ensure the pipeline is written in a way that doesn't generate a ton of memory traffic with intermediate arrays; I haven't checked to see what the library they show does.) Presumably if I'm writing this in Go I have some other reason for wanting to do that, like having some non-trivial concurrency desire (using concurrency to handle a newline-delimited JSON file was my major use case, doing non-trivial though not terribly extensive work on the JSON).
While this may make some people freak, IMHO the real point of "errors as values" is not to force you to handle the errors in some very particular manner, but to make you think about the errors more deeply than a conventional exceptions-based program typically does. As such, it is perfectly legal and moral to think about your error handling and decide that what you really want is the entire program to terminate on the first error. Obviously this is not the correct solution for my API server blasting out tens of thousands of highly heterogeneous calls per second, but for a shell script it is quite often the correct answer. As something I have thought about and chosen deliberately, it's fine.
package main
import (
"github.com/bitfield/script"
)
func main() {
script.Stdin().Column(1).Freq().First(10).Stdout()
}Now you can run this:
cat file.txt | ./goscript.sh -c 'script.Stdin().Column(1).Freq().First(10).Stdout()'
#!/tmp/goscript.sh
script.Stdin().Column(1).Freq().First(10).Stdout()
chmod 755 topten.sh
echo "one\none\ntwo" | ./topten.sh _, err := script.Foo(...).Bar(...).Stdout();
if err != nil {
log.Fatal(err)
}
I don't see it as a lot different to bash scripts with -e and pipefail set, which is generally preferable anyway.
Plenty of go code does
if err != nil {
return nil, err;
}
(and under -e plus pipefail, "error reported to stdout followed by aborting" is pretty much what you get in bash as well, so I'm unconvinced it's actually going to be harder to debug)
That's hardly the whole point of using Go.
The friendlier syntax (and in this case DSL) is an ever bigger point.
In any case, you can trivially get at the error at the point it occured:
n, err := script.File("test.txt").Match("Error").CountLines()
package main
import (
"github.com/bitfield/script"
)
func main() {
_, err := script.Stdin().Column(1).Freq().First(10).Stdout()
if err != nil {
log.Fatal(err)
}
}
cat file.txt | ./goscript.sh -c 'script.Stdin().Column(1).Freq().First(10).Stdout()'
Details here: https://til.simonwillison.net/bash/go-script
For scripting I want something that I can be fast and messy in. Go is the opposite of that.
It's ok, a language doesn't have to be good at everything.
Also, I like being able to make modifications on the fly, so doing something in Ruby, I can just open the file make adjustments and I am done. With Go, I have to compile it and move it back into my path which is really tedious.
Many people love to smirk and say "just learn bash properly, duh" but that's missing the point that we never do big projects in bash so our muscle memory of bash is always kind of shallow. And by "we" I mean "a lot of programmers"; I am not stupid, but I have to learn bash's intricacies every time almost from scratch and that's not productive. It's very normal for things to slip up from your memory when you're not using them regularly. To make this even more annoying, nobody will pay me to work exclusively with bash for 3 months until it gets etched deep into my memory. So there's that too.
I view OP as a good reminder that maybe universal-ish tools to get most of what we need from shell scripting exist even today but we aren't giving them enough attention and energy and we don't make them mainstream. Though it doesn't help that Golang doesn't automatically fetch dependencies when you just do `go run random_script.go`: https://github.com/golang/go/issues/36513
I am not fixating on Golang in particular. But IMO next_bash_or_something should be due Soon™. It's not a huge problem to install a single program when provisioning a new VM or container either so I am not sure why are people so averse to it.
So yeah, nice article. I like the direction.
EDIT: I know about nushell, oilshell and fish but admittedly never gave them a chance.
"Please use the original title, unless it is misleading or linkbait; don't editorialize." - https://news.ycombinator.com/newsguidelines.html
.... but there are just soooo many little annoyances / inconveniences which turn me off.
- No Optional Parameters. No Named Parameters. Throw us a bone Rob Pike, it's 2023. Type inferred composite literals may be an OK compromise.. if we ever see them: https://github.com/golang/go/issues/12854
- Unused import = will not compile. Unused variable = Will not compile. Give us the ability to turn off the warning.
- No null safe or nullish coalescing operator. (? in rust, ?? in php, etc.)
- Verbosity of if err != nil { return err; }
- A ternary operator would be nice, and could bring if err != nil to 1 line.
- No double declarations. “no new variables on left side of :=” .. For some odd reason “err” is OK here... Would be highly convenient for pipelines, so each result doesn't need to be uniquely named.
I'd describe Go as a "simple" language- Not an "easy" language. 1-2 lines in Python is going to be 5-10 lines in golang.
Note: Nim has most of these..
Shameless plug: this is why I built Risor.
https://github.com/risor-io/risor
Keep in the Go ecosystem, retain compatibility with the Go programs you already have, but have a much more concise scripting capability at your disposal.
This is Typescript, but you have language complaints, and it will build binaries.
export LC_ALL=C
awk '!a[$1]++' access.log|head
When this happens, one can split access.log into pieces, process separately then recombine.
But that's more or less what sort(1) does with large files, creating temporary files in $TMPDIR or other user-specified directory after -T if using GNU sort.
There was a way to eliminate duplicate lines from an unordered list using k/q, without using temporary files but I stopped using it after Kx, Inc. was sold off and I started using musl exclusively. q requires glibc.
For example, something like
#!/bin/sh
# usage: $0 file
echo "k).Q.fs[l:0::\`:$1];l:?:l;\`:$1 0:l"|exec q >null;
The other approach I use to avoid temporary files is to just put the list in an SQL database, add a UNIQUE constraint, and update the database.
/*usr/bin/env go run "$0" "$@"; exit $? #*/
for example:
import { jsh, cat, grep, PipeOutput } from 'jsh;
// type PipeOutput = { stdout: ReadableStream, toString: () => Promise<string>, extra: Record<string,any> }
function countLines(input: PipeOutput, argv: string[]): PipeOutput {
// ...
}
const textToLookFor = process.argv[1]
const output: PipeOutput = jsh`${cat} file.txt | ${grep} ${textToLookFor} |
${countLines}`
console.log(output.toString())https://github.com/risor-io/risor#quick-example
Stay in the Go ecosystem, but gain pipes, Python-like f-strings, and more.
(I'm the author)
gron | rg
FTW
<time class="dt-published date-callout" datetime="2022-02-21" pubdate="">
<div class="date-wrapper">
<span class="month">Feb </span><span class="day">21</span>
</div>
</time>
FYI Go has Compact now:
Scripting with Go - https://news.ycombinator.com/item?id=30641883 - March 2022 (66 comments)