Scripting with Go
bitfieldconsulting.com
bitfieldconsulting.com
I read the intent of the original package as a bit different than how it’s presented here; a pain point in using go for systems programming on Posix systems with GNU tooling is interacting with all the other really excellent tools there. There’s an awful lot of cruft to go spawn a shell script in Go without this lib.
So, I cannot imagine using this to replace a short-ish bash script, it’s just too heavyweight to add go development into the workflow. But, I can easily imagine using this when I want to do some simple-to-medium-scale pipelined text processing from the middle of a go program.
Shell scripting is far from perfect, and sometimes it does take a while to figure out how to do something that would be easy in another language - but very often, she'll scripting is "good enough".
I’ve written plenty of bash scripts to automate common tasks. Colleagues won’t open them up and extend or maintain them (some developers won’t touch bash). But if they have an easy to use, well documented interface, they will happily use them to be productive.
The problem with Python is as you describe. It’s not always guaranteed and if you want everyone’s environment on the same page, you then have to start faffing with virtual envs and/or version managers.
This is totally fine on projects written in Python as a whole. But, otherwise, adding complexity to the tool chain is a tough sell to the team.
If things get hairy in bash, it’s probably better to extract those parts in the project language and call them from bash. Your build will already be half way there to accommodate this vs introducing a scripting language.
Not only that, maintaining those programs involves less context switching. Although mainstream languages blur with enough experience, there is still overhead between juggling idioms, syntax and what not in, say, Python and Go. I’m not perfect: if I’ve been working all day in one language, writing another is like a smack in the face. Sometimes I’ll forget semicolons, or include them. Sometimes I’ll use camel case when really the style in that language is snake case. Not a big deal but it is jarring.
There is still context switching with bash but if you use a terminal-based workflow, that does have less of an impact.
https://stackoverflow.com/questions/5694228/sed-in-place-fla...
Yeah, this is where I see it.
I have a cross-platform utility in customer use that replaces a bash script or a .bat file that did a simple job with curl and a database client but was tricky for customers to configure and deploy; I replaced it with an interactively-configurable go command.
In the middle of it is the replacement pipe; this library would help here. Might use it in the rewrite.
The biggest benefit of the shell is its clearly defined input and output (and error) interfaces. Most programming languages can read from and write to stdin, stdout, and stderr.
Why not use it and stick to KISS, replacing one cumbersome POSIX utility at a time, suites for the task? Then you don’t need to chain methods using less idiomatic code. But then you wouldn’t need these kind of libraries either.
maybe that's the compelling use of scripting with a statically typed, thus compile-time at least partially low-hanging-fruit-error-checked, language?
IMHO writing procedural style code with lots of if, loops, etc. in the shell can quickly turn into an anti-pattern. Try to stick to simple functions that are chained together in pipelines. The only loop is typically one that processes arguments and that's it.
Bash is incredibly less complex than Javascript and there is such a resource: the "Bash guide" [1] and "Bash pitfalls" [2] are both excellent resources that teach you how to use Bash properly.
Oh, but which version, 2 or 3?
Bash availability may not be guaranteed, but a POSIX shell of some variety is.
Shellcheck has dramatically increased the ease by which you can write decent scripts before ever running them. But I also have never seen any language that anyone could master without a year or more of constant use.
For example, you could in your scripting language use `find` to search for files in a folder and do something with them, but why do that when your language of choice almost certainly has globbing capabilities? You could grep a file for a line, but why do that when you can use your language's inbuilt regex system?
At least from the scripting side, the reason I tend to push more towards using the language and less towards using external processes is because most scripting languages can do what those external processes do in one go.
Perhaps it would make more sense if I were better at defining scope :)
I did manage to make a Windows batch file that replicated the functionality of a linux/mac bash script, but configuring it was no fun for customers on any platform, and then there were the utilities themselves to deploy.
The replacement binary has a very small platform-dependent aspect, and I am not held back by the limits of batch files when trying to achieve feature parity.
It might be doable to deploy a powershell script, but then there's installation work to do on the unix side instead.
This is the approach that I took with murex. It comes with a stack of builtins for json, jsonlines, yaml, toml, csv, sqlite3, and others. So you have all the power of data types and intelligent management of structured data but also works with regular POSIX CLI commands without any additional boilerplate code when swapping between murex code and coreutils (et al). So one could say I’ve spent a fair amount of time studying this point you’ve raised :)
(Don't be afraid to self promote!)
One obvious benefit over what's suggested in the article is that you can use it interactively first, with autocomplete and such goodies, and transition smoothly to a script later.
I hope the go devs will reconsider. I'd love to be able to use go for scripting. But as it stands, it's a sad state of affairs because you have to rely on hacks.
> The biggest impediment is the ceremony for subprocessing out
Not sure I fully understand what you mean with this, but if I'm understanding it right, gorun attempts to solve this by doing `syscall.Exec()` on the compiled binary. Combined with slight variation on the "comment" hack (see examples in gorun's readme), signals get fully directed your binary.
// 2>/dev/null ; exec gorun "$0" "$@"
package main
// rest of the code go run github.com/mikefarah/yq/v3@3.4.1
#!/usr/bin/env -S go run github.com/mikefarah/yq/v3@3.4.1 r
---
some:
yaml: here
./test.yaml some.yaml
testClojure's data first model + conveniences like the -> and ->> macros make simple data pipeline scripts a joy to write.
babashka of course suffers from the "it isn't everywhere" problem that is mentioned regarding python.
Using a third-party libraries as core part of your infrastructure (ci/cd scripts, provisioning, automation) implies a greater risk to potential security issues, "framework tax" ie. having to comply, learn, document, debug its custom APIs, having to deal with potential limitations, issues that either needs to be fixed upstream or result in the library being forked and therefore maintained in house. I would rather either put together a set of bash commands or - if the problem entails a more comprehensive endeavour with greater complexity - put together a in-house tool/library where i can make the right compromises from day one
Unix decide to "simply" creating a system with a system language and an user system with an user language, the shell. I do not like much unix approach after having used Emacs a bit, but I do understand it. On contrary I always fails to "craft scripts" in "real" programming languages no matter what. I've tried in the past to "go Perl", "go Python", "go TCL", yes I can write scripts with them, I have written some etc but if I need something quick I go for the shell, zsh to be more precise (tried others, all failed at a certain point, from bash to elvish passing through oil shell and few others) or being in Emacs (EXWM is my WM/DE) Emacs itself depending on the case.
I read various similar article for a language or another but in the long run see no colleagues really choose a programming language behind shell itself...
https://github.com/oilshell/oil/wiki/Internal-DSLs-for-Shell
It looks like this particular one relies on method chaining
script.Args().Concat().Match("Error").First(10).Stdout()https://blog.cloudflare.com/using-go-as-a-scripting-language...
p := Exec("man bogus")
p.SetError(nil)
output, err := p.String()
fmt.Println(output)
man bogus
n, err := object
.Process1(arg1, arg2)
.Process2(true)
.Process3("foo")
https://github.com/89z/googleplay/blob/v1.8.1/header.go#L126
Quote the author: "The script library is implemented entirely in Go, and does not require any external userland programs." Sorry but that's not scripting anymore. We all like using userland programs! it is The Way.
Most programs tell you why they failed on stderr. Seems like stderr is lost when a pipe component fails. Strangely stderr and stdout are conflated in the pipe structure - there is no way to get stderr!
You just get the numeric exit status.
IO redirection appears to be missing.
Difficult to use in production.
It seems like a shame that this kind of power and expressiveness is reserved only for generating text.
At one time jQuery was popular and this seems like a similar thing, but for files?
It's small enough that if you're worried about "other people's code" you could fork it and maintain it yourself.
I agree with them.
One of the problems I have with things like LINQ or Promises is that they are immensely powerful but can be really difficult to debug when things go wrong because of implicit errors within the pipeline bubble up the stack without providing location context. While the standard boilerplate around errors in Go is annoying, I actually prefer it for two reasons:
1. It provides exactly where things went wrong while looking at an exception trace, and 2. From a readability perspective, it is much easier to understand what the author of the code meant to do and, more importantly, what the follow-on error s mean. Sometimes, errors aren't actually errors.
Either way, OCaml/Haskell/F#'s approach with match expressions and the `Result` type is the best way to deal with this, IMO. You get the best of both worlds: explicit declaration with a very expressive (triangular-like) "shape" to the code.
So something like this:
var data string = script.File("test.txt").Match("Error").CountLines()
var foundLines int32 = 0
var numLines int32 = match file.Open("test.txt") with
| Error e -> return e
| Ok f -> match f.Readlines() with
| Error e -> return e
| Ok lines -> match lines with
| /Error.*/ -> foundLines++
| _ -> // do nothign
return foundLines
foundLines := 0
lines, err := ioutil.ReadFile("test.txt")
if err != nil {
return 0, err
}
for _, l := range lines {
re, err := regexp.MustCompile(`Error.*`)
if err != nil {
return 0, err
}
matches := re.FindStringSubmatch(`Error.*`)
if len(matches) > 0 {
foundLines++
}
}
return foundLines, nil
As far as the library itself, I personally wouldn't use something like this, though I see the appeal. I turn to statically-typed languages like Golang when a Bash script becomes too kludgey to stand on its own (usually when I need to begin relying on mild data structures) and when I care about type safety and portability more than what Ruby or Python can give me. When I'm writing a Go program, I want something that's testable that I know can work anywhere. With Ruby or Python, unless I'm distributing the script as a Docker image, I have to worry about versions, environments, etc., none of which are pleasant.
However, writing the error boilerplate is annoying, and I can see developers who want to write something quick but spend 99% of their time in Go using this to get something done with a tool they know well. I've seen similar things in the Java world; hell, that's the reason why Groovy and Kotlin exist!
TL;DR: The "wrong thing" in the Bitfield article isn't necessarily wrong; their library is useful, but niche; all hail pattern-matching and the Result type.
foo: Optional[Foo]
if foo is None:
return some_error_handler()
bar = foo.qux() # the above guard "upgrades" O[Foo] to Foo
Also specifically in Python, the type system is kind of weak (especially those bundled with Pycharm), and a lot of functional operations which ought to be more strongly typed end up with their types erased for whatever reason. In particular functools.partial seems to discard type information more than I'd like. The Result python package [3] doesn't run into this problem near to the same degree.
1] https://martinfowler.com/bliki/FluentInterface.html
2] I find this term frustrating not only because it's infamously ineffable, but also because there appear to be disagreement whether Rust's containers are actual monads or not, leading me to dub what I call the "Engineer's Monad" which is a generic container you can map and flatmap over, even if isn't strictly an Applicative.
You can't emulate Result[Result[T]] with a simple Union.
The space of "shell-like libraries", not least of which is shell itself, is so rich it's hard to beat out everything else on its own merits. But having something as easy as shell that integrates back into your Go ecosystem is very convenient.
And I'm sure similar things are true for the other libraries in other languages.
So I would personally not present this as "here's something awesome Go can uniquely do", but, "if you already have Go, be aware of this tool/technique".