Atuin Desktop: Runbooks That Run
blog.atuin.sh
blog.atuin.sh
You can have a plaintext file which is also the program which is also the documentation/notebook/website/etc. It's extremely powerful, and is a compelling example of literate programming.
A good take on it here: https://osem.seagl.org/conferences/seagl2019/program/proposa...
The idea has a lot of merit. We even gave a talk about it in JupyterCon Paris 2023 - https://www.youtube.com/watch?v=TUYY2kHrTzs
When you have executable code in the documentation, folks want to follow PR-review workflow with the docs as well - which is a bit more team investment than editing a wiki.
Good luck!
(Hosted notebooks with IAM integration.)
Curious how long ago were you at AWS? For context, I spent the last few years in AWS working on an internal platform service whose entire purpose was to reduce operational toil by helping you codify your operational runbooks and execute them safely and automatically. Atuin Desktop is similar to that service in some sense but that service just offered much more features.
Genuine question. Not familiar with this company or the CLI product.
Usually, I feel like Jupyter gives both worlds—- flexible scripting and support for os commands (either through !/% or even os.system()
For me, pipenv/pyenv/conda/poetry/uv/dependencies.txt and the invitable "I need to upgrade Python to run this notebook, ugh, well, ok -- two weeks later - g####m that upgrade broke that unrelated and old ansible and now I cannot fix these fifteen barely held up servers" is pure hell.
I try to stay away from Python for foundational stuff, as any Python project that I work on¹ will break at least yearly on some dependency or other runtime woe. That goes for Ansible, Build Pipelines, deploy.py or any such thing. I would certainly not use Jupyter notebooks for such crucial and foundational automation, as the giant tree of dependencies and requirements it comes with, makes this far worse.
¹ Granted, my job makes me work on an excessive amount of codebases, At least six different Python projects last two months, some requiring python 2.7, some requiring deprecated versions of lib-something.h some cutting edge, some very strict in practice but not documented (It works on the machine of the one dev that works on it as long as he never updates anything?). And Puppet or Chef - being Ruby, are just as bad, suffering from the exact same issues, only that Ruby has had one (and only one!) package management system for decades now.
Exactly what I was looking for, thanks!
Love runnable documentation. We don't have enough of it.
Immediately after signing in, I get an email from you. Serious uncool and an instant "nope" from me after that.
We recently started using https://marimo.io/ as a replacement for Jupyter notebooks, as it has a number of great improvements, and this seems like a movement in a similar direction.
If you want to record a runbook, then record a runbook. You can do that a million ways. Text file, confluence doc, screen recording, shell script, etc. People already don't do that; they're not gonna suddenly start doing it more because your UI is fancier.
Personally, I don't want to sit around all day writing code (or docs) to try to get the system to be like X state. I want to manually make it have X state, and then run a tool to dump the state, and later re-run the tool to create (or enforce) that state again. I do not want to write code to try to tell the computer how to get to that state. Nor do I want to write "declarative configuration", which is just more code with a different name. I want to do the thing manually, then snapshot it, then replay it. And I want this to work on any system, anywhere, without dependence on monitoring a Bash shell for commands or something. Just dump state and later reapply state.
Like, Terraform has always sucked because there was no way to dump existing resources as new code. So a team at Google made a tool to do it (Terraform-er). If Terraform had already had that feature, and if it didn't rely on having pre-existing state to manage resources, that would be like 90% of the way to what I'd want. Just dump resources as code, then let me re-run the code, and if I want I can modify the code to ask me for inputs or change things. (People think of Terraform as only working on Cloud resources, but you could (for example) make an Ubuntu Linux provider that just configures Ubuntu for you, if you wanted)
> Just dump state and later reapply state
is necessarily declarative.
> Just dump resources as code,
What is the code for this resource?
VM foo1
Memory 16GiB
Network mynet1
That's a solved problem. Anything that you use on a computer that controls a resource, can uniquely identify said resource, through either a key or composite key. This has to be the case, otherwise you could create things that you could never find again :) (Even if you created an array of things with no name, since it exists as an item in a list, the list index is its unique identifier)
Taking Terraform as example again, the provider has code in it that specifies what the unique identifier is, per-resource. It might be a single key (like 'id', 'ASN', 'Name', etc) or a composite key ( {'id' + 'VPC' + 'Region'} ).
If the code you've dumped does not have the unique identifier for some reason, then the provider has to make a decision: either try to look up existing resources that match what you've provided and assume the closest one is the right one, or error out that the unique identifier is missing. Usually the unique identifier is not hard to look up in the first place (yours has a composite identifier: {VM:"foo1", Network:"mynet1"}). But it's also (usually) not fool-proof.
Imagine a filesystem. You actually have two unique identifiers: the fully-qualified file path, and the inode number. The inode number is the actual unique identifier in the filesystem, but we don't tend to reference it, as 1) it's not that easy to remember/recognize an inode number, 2) it can be recycled for another file, 3) it'll change across filesystems. We instead reference the file path. But file paths are subtly complex: we have sym-links, hard-links and bind-mounts, so two different paths can actually lead to the same file, or different files! On top of that, you can remove the file and then create an identically-named file. Even if the file had identical contents, removing it and creating a new one is technically a whole new resource, and has impact on the system (permissions may be different, open filehandles to deleted files are a thing, etc).
So what all of us do, all day, every day, is lie to ourselves. We pretend we can recognize files, that we have a unique identifier for them. But actually we don't. What we do is use a composite index and guess. We say, "well it looks like the right file, because it's in the right file path, with the right size, and right name, and right permissions, and (maybe) has the right inode". But actually there's no way to know for sure it's the same file we expect. We just hope it is. If it looks good enough, we go with it.
So that's how you automate managing resources. For each type of resource, you use whatever you can as a unique (or composite) identifier, guesstimate, and prompt the user if it's impossible to get a good enough guess. Because that's how humans do it anyway.
I don't think this is accurate. Terraform operates against a state snapshot, which is usually local but can also be remote. But it has several mechanisms to update that state, based on the current status of any/all defined resources, see e.g. `terraform refresh` (https://developer.hashicorp.com/terraform/cli/commands/refre...) -- and there are other, similar, commands.
> But for a moment I'll assume you're talking about the other problem with configuration management tools, which is "which of the existing resources do I actually want to exist or modify?"
I'm not really talking about that specific thing, no. That problem is one of uncountably many other similar sub-problems that configuration management tools are designed to address. And, for what it's worth, it's not a particularly interesting or difficult problem to solve, among all problems in the space.
If you have a desired state X, and an actual state Y, then you just diff X and Y to figure out the operations you need to apply to Y in order to make it end up like X. Terraform does this in `terraform plan` via a 3-way reconciliation merge/diff. Pretty straightforward.
> you just want an item to exist; if it does exist, great, if it doesn't exist, you create it
It's not as simple as whether or not an item should exist. Being able to uniquely identify a resource is step one for sure. But a single resource, with a stable identifier, can have different properties. The entire resource definition -- identifier, properties, and everything else -- is what you type and save and commit and push and ultimately declare as the thing you want to be true (X). That's not code, it's state (definitions). Code is what's executed to diff that declarative state (X) against actual state (Y) to produce a set of delta operations. Or, it's those delta operations themselves.
> If the code you've dumped does not have the unique identifier for some reason, then the provider has to make a decision: either try to look up existing resources that match what you've provided and assume the closest one is the right one...
First, you "dump" state, not code. More importantly, no configuration management system would ever take one identifier and "guesstimate" that it should match a different identifier, because it's "close", whatever that means.
> or error out that the unique identifier is missing. Usually the unique identifier is not hard to look up in the first place (yours has a composite identifier: {VM:"foo1", Network:"mynet1"}). But it's also (usually) not fool-proof.
I really don't understand what you mean, here, nor do I understand your mental model of these systems. It's certainly not the case that my example VM has the composite identifier {vm:foo1 network:mynet1}. The identifier is, intuitively, just foo1. Even if we were to say the identifier were an object, the object you propose is missing the memory size. But more importantly, changing the foo1 VM from network:mynet1 to network:othernet2 probably should not have the effect of destroying the existing VM, and re-provisioning a brand new VM with the new network. Sometimes configuration changes require this kind of full teardown/spinup, but these conditions are generally rare, and all modern configuration management tools avoid this kind of destructive work whenever possible and most of the time.
> So that's how you automate managing resources. For each type of resource, you use whatever you can as a unique (or composite) identifier, guesstimate, and prompt the user if it's impossible to get a good enough guess. Because that's how humans do it anyway.
Just to reiterate, I'm not aware of any configuration management tool that "guesstimates" when making changes in this way. For good reason.
If Terraform wasn't completely idiotic, it could have just checked if it existed in the planning stage. If Terraform was even mildly helpful, it would have suggested to the user at either plan or apply time that the security group already exists, and do you want to manage that with your code? But it doesn't do those things, because it's a completely dumb-ass design.
> I'm not aware of any configuration management tool that "guesstimates" when making changes. Thank God.
Many of them do. Ansible does, Puppet does, Terraform does. They have to, for the same reason as my filesystem example: it's often impossible to know that a resource is unique, because there aren't actually unique identifiers. My definition of "Guesstimation" is specifically "using the identifiers you have available to select an entry from a list of potential options with the closest match". Ansible does this all the time. Puppet and Terraform do this for every provider that doesn't have a totally unique identifier (there basically are no totally unique identifiers, as I pointed out in my filesystem example)
It seems to me that your frustration with Terraform being "completely idiotic" is ultimately frustration with the underlying design model.
> If you write configuration to create a Security Group named "foobar",
That configuration is a declaration: a security group named "foobar" should exist, with the declared properties.
> and do a `terraform plan`, it will say it's about to create "foobar".
That plan would be based on the most recent snapshot of the target "outside world" resources, which, if you haven't synced them recently (or at all) would probably be empty, resulting in `terraform plan` proposing to create foobar afresh.
> When you go to apply, it will error out, saying "foobar already exists"
Sure, which should hopefully make sense. You've declared a resource locally, and asked Terraform to "make it so" basically. But that resource is in conflict with an identical remote resource. You can `terraform refresh` or sync or whatever, to pull down the current relevant remote resource state locally, and then operate from there. Or you can manually blow away the remote foobar and retry. Or etc.
But this kind of situation is not common. Terraform assumes and expects that the declarations (and state) it has access to locally is an authoritative source of truth for what the target remote system(s) should be. The config files define what should be running in AWS, not the other way around.
It's fine if this isn't a fit for your use cases, but I don't think that means the entire tool is stupid or whatever. It just means it's not for you.
It essentially still is.
Unless the Dockerfiles are kept secret, any container can be replicated from the given Dockerfile. Barring extreme (distro/system/hardware)-level quirks, a Docker container should be able to run anywhere that Linux can.
I imagine with a lot of discipline (no apt update, no “latest” tag, no internet access) you can make a reproducible docker file…. But it is far from normal.
Beware, I am definitely not claiming those are easy to use in general. Just that you can get to reproducibility using them more reliably and maybe easier than with docker.
Incorrect. Step one of reproducibility is "disable unconstrained downloading from the internet". Docker does the opposite.
If you want a reproducible Docker image, you're on your own. For example, the most common problem is that many build scripts out in the wild download stuff willy nilly without verifying anything. I've seen NPM package post install scripts do the craziest things. Catching all of that is harder than most would give credit for at first glance, considering that tons of build scripts are written in Turing complete languages. Help from tooling is essential.
When you have to fight the tool to achieve reproducibility, not choosing to do so isn't "using a tool poorly." It's simply using the tool as is. Especially when the vast majority of Dockerfiles out there happily run something along the lines of `apt install foo`, again, without verifying anything.
However the payoff is rather significant so if you can temper that cost a bit and make it less inconvenient to achieve then you have a winning solution.
I have cooked this up based on Bazel, rules_oci and rules_distroless: https://github.com/josephglanville/images Specifically this file is a busybox based image with some utilities included from a Debian snapshot: https://github.com/josephglanville/images/blob/master/toolbo...
More difficult than Dockerfile? Sure. However better in pretty much every way otherwise including actual simplicity.
But mysteriously, people say that Docker is reproducible because, uh, you can download gigabyte-sized binary blobs from the Docker registry. I wonder, what's not reproducible by that metric?
Docker images may be portable compared to binaries targeting traditional FHS distros. But it's not reproducible whatsoever.
Docker images (not files) help with the run time consistency .
Docker (files) barely scratch the surface of build reproducibility. Most applications depend on the distribution package manager ( apt, apk etc) and language package manager (npm, cargo, etc), both sets of them have various challenges in consistent dependency resolution.
In addition build steps might have ordering challenges RPC calls to remote services no longer running and so on.
Anyone trying to to build a docker image from 10 years back experiences this problem
It's basically the best and easiest "I am documenting how it works now" thing without any arcane "works on my machine" quirks I have yet found.
So I'm still agreeing here that it's a very good approximation of this idea.
Real reproducability is miles better, but usually cannot be formulated in a ~20 line single file "recipe". (and before anyone mentions Nix.. no, there's so much inherent complexity involved, that doesn't count like "apt-get install docker && docker build ."
Dockerfiles are basically this, but with a file documenting the different steps you took to get to that state.
Step 1. You write a Dockerfile. You build it. You test with it. Hey, it works! You push it to production.
Step 2. Years later, you need to patch that container in production. So you change the Dockerfile, rebuild, and re-test.
Step 3. Uh-oh! The tests don't work anymore! What's going on? I changed one line in the Dockerfile but now the app isn't working (or perhaps the build isn't). What's going on?
What's going on is a reproducibility failure. Just having the instructions (or what someone thought was the instructions, or what they were years ago) isn't enough to ensure you get the same results a second time. A million little things you didn't think of may change between builds (and the more time in-between, the more things change), and they may end up breaking things. Without a truly reproducible build, you are likely to have problems trying to rebuild with just the instructions.
That's why with Docker containers, we rely on build artifacts. The container you had two years ago? However that was built, with whatever weird combination of arguments and files-on-the-internet-being-pulled-at-build-time and everything else, it was built successfully. If you want to be really sure you patch it correctly, you pull the old container image (not Dockerfile), modify that, and push that to production as a new container. No rebuilding, just patching. This avoids reproducibility failures.
That same idea is why you'd want to just download a blob and later re-apply it.
The blob was the state of things when it was working. If you tried to just write down the instructions to replicate it, it's likely you'd either 1) get it wrong (it was only working because of some unrelated changes somebody else made and forgot about) or 2) you'd get a reproducibility error.
So "the blob" I'm talking about doesn't have to be a literal binary blob. It could be whatever, we're talking about a theoretical idea here. It could be layers like a container, or metadata in a JSON file, or configuration/code that gets auto-generated, etc. I don't care what it is. It just has to describe the state as it was when X was working. It's then up to the tool to figure out how to get back to that state.
People already write this as "declarative code" for configuration management tools to do the same thing. I'm saying, I don't want to have to write the code. Just dump it out for me.
Can you expand on this?
This will lead whatever steps you've recorded to 1) stop working on the existing "local" machine, and 2) be incompatible with other people's "local" machines. So the instructions have "rotted".
To avoid this you could write Ansible/Puppet to install and configure all the same tools, but that will break too over time too, and be a maintenance hassle. The only reliable solution is to use containers to run it all in; that guarantees the same version of everything. But at that point it's not really "running locally" anymore, it's running in the container, which is sort of its own bag of issues.
At that point you might as well have a SaaS tool to run your runbook in a cloud environment in containers or something, as that's way easier to set up and manage than either ansible/puppet, or Docker on everyone's machine (there's still a million tech "engineers" out there who don't understand containers).
ALL databases had their startup configuration parameters defined per instance across datacenters.
Furthermore SRE had tools to rate limit connections or restart of a database so that it was not overwhelmed by incoming connection requests. We also built tools to do fine grained definition of load balancers and what services were behind them to be able to redirect traffic on the fly and then reset to the original mappings once connectivity of the LBs was resolved.
These tools weee centralized and available to both SRE and senior system administrators.
These things evolved to new tools which accommodated a private cloud, then docker then Kubernetes. I left prior to kubernetes implementation.
- I am on a team that oversees a bunch of stuff, some of which I am very hands-on with and comfortable with, and some of which I am vaguely aware exists, but rarely touch
- X, a member of the latter category, breaks
- Everyone who actually knows about X is on vacation/dead/in a meeting
- Fortunately, there is a document that explains what to do in this situation
- It is somehow both obsolete and wrong, a true miracle of bad info
So that is the problem this is trying to solve.
Having discussed this with the creator some[1], the intent here (as I understand it) is to build something like a cross between Jupyter Notebooks and Ansible Tower: documentation, scripts, and metrics that all live next to each other in a way that makes it easier to know what's wrong, how to fix it, and if the fix worked
[1]Disclosure: I help mod the atuin Discord
How does Atuin solve that problem? It seems to me that inaccurate and obsolete information can be in an Atuin document as easily as in a text document, wiki, etc., but possibly I'm not seeing something?
I believe the intent is that you get bidirectional selective sync between your terminal and the docs, so that if what's in the docs is out of date or wrong, then whatever you did to actually fix things can be synced back to the docs to reduce the friction of keeping the docs updated.
I'm cautiously curious about something like this, although I haven't tried it personally.
The idea seems interesting to me just cause I do not really like terminals and having something more visually appealing and with better history and comments is an improvement though I am also not sure if Atuin is best way to achieve all of that.
However I don't see how it solves the obsolete or wrong documentation thing. You still have to make sure the runbook is correct, if it's not you've got the exact same problem.
Having a centralised place for all your scripts is an advantage with inline docs. But then this is a local desktop version...
Thus “Runbooks That Run.”
Some people just like a particular workflow or tooling flow and build it really. Maybe it works for enough people to have a viable market, maybe not.
I am just using a PHP deployment process for no reason other than feeling like it for personal projects and it handles 60% of the work without me needing to do anything. But any runbooks for it are tasks built into the tool and in the same git repo for the entire server deployment. I'm not gonna put it in some random place or a shell script that I need to remember separate commands for.
Code, for programmers, is inherently self-documenting if you keep a simple functional style without any complexity with comments on the occasional section that isn't just "Create a MySQL user, roll the MySQL user's password, update the related services with the new password/user combination, remove the old user that the fired employee has credentials to on the off chance we failed to block them at the VPN" kind of stuff.
That's entirely different to what's being desired by GP.
> > My dream tooling is for every tool to have an terminal interface so that I can create comprehensive megabooks to get all the context that lives in my head. i.e. jira, datadog, github etc, all in one pane.
My perspective on this is essentially having jira/datadog/github/etc be pluggable into the CLI, and where standard bash commands & pipes can be used without major restrictions. (Something akin to Yahoo Pipes)
MCP is highly centered around LLMs analyzing user requests & creating queries to be run on MCP servers. What's being desired here doesn't centralize around LLMs in any sense at all.
We build logic once and make it automatically accessible from any of these consumption methods, in a standardized way to our clients, and I am indeed piping some of these directly in the CLI to jq and others for analysis.
and plenty of unofficial ones
World of API...
Tauri means we can reuse a lot of the Rust we already have, easily do the systems stuff we need, and have something light + fast. Elixir has been awesome and makes a realtime sync backend easier
Not currently open source while it's under heavy early development, we will be opening up the desktop app later on
(nb: system web views are very inconsistent, so they're considering adding a Chromium renderer, which will bring everything full circle)
we've found they're generally ok between mac/windows, with some issues on Linux. Nothing insurmountable, however.
Anything super complex (terminals, charts) we can render with canvas or webgl anyway
This leaves room for stuff like the Functional Software License.
Software freedoms exist as a concept for a reason, not just a bullet point to get people to click a download link that doesn’t even include source anyway.
I call such projects “open source cosplay”. It’s an outfit you put on for conferences, then take off when back at the office working on the nonfree valuable parts.
The irony of this purist mindset is that it's actually very corporatist, big-tech, and proprietary in its implications. If open source devs are discouraged by the culture from building products and making a living independently, it means that the only people who can devote significant time to open source are employees of established companies (who themselves often sell closed source proprietary products) and people who are wealthy enough to work for free. Is that the world you want?
Why would I waste my time releasing any of my projects for free when people will attack me and call me a poser anyway
Might as well charge people money, who by the way will actually be grateful to do so, that try to keep up with the open source community's purity treadmill
It’s ok if you don’t believe in software freedoms, but you shouldn’t pretend to be someone who does by releasing some software that respects users’ software freedoms. It’s deceptive.
Either you care about software freedoms, or you don’t. If you don’t, why are you releasing any software under free software licenses? If you do, why are you releasing any nonfree software?
Also, do you have a single bit of backing data to suggest that your first sentence is true? I don’t believe that it is. It seems to me there is more free software than ever before.
As long as the different parts are clearly marked/indicated as such, why would you impose such a ridiculous standard? In your world, if a company makes 99% of their software GPL, and then releases some proprietary tool, they're suddenly being deceptive? Would you prefer to just lose the 99%?
In the example case, they plainly do not, otherwise they would not subject their users to nonfree software.
I don’t think wishing vendors to be honest is a ridiculous standard.
IMHO the real reason is that the threat of hostile/competing forks has gone up. It used to be gauche at best, evil at worst, to take somebody's open source code and compete with them, but increasingly the landscape is changing. I think that's the real problem, and IMHO the answer to that is the AGPL, not to go proprietary.
Using C# as main language this allowed us to have runbooks using code shared as nuget package and so being able to interact with our own APIs and applications as any other code that runs in production.
Not the best experience to review but it worked for us.
Document it, version control it, share it, and (semi)automate it. There's no reason to type or copy-paste important commands directly into the terminal.
There is a response elsewhere in comments[1] which claims that this is trying to fix the problem of bad documentation, but this has the same fundamental problem. If you a) are responsible for fixing something, b) are unfamiliar with it, and c) the "fixing resources" - whether those are scripts, documentation, or a Runbook/Workflow - you were provided with by the experts are out-of-date; you're SOL and are going to have to get to investigating _anyway_. A runbook and a script are just different points along the spectrum of "how much of this is automated and how much do I have to copy-paste myself?"[2] - both are vulnerable to accuracy-rot.
[0]: https://www.warp.dev/warp-drive
[1]: https://news.ycombinator.com/item?id=43766842
[2]: https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...
Because everything is a start-up now.
this project appears to be intended for operational documentation / living runbooks. it doesn't really seem like the same use case.
As an on-the-fly debugging tool, they're great: you get a REPL that isn't actively painful to use, a history (roughly, since the state is live and every cell is not run every time) of commands run, and visualization at key points in the program to check as you go your assumptions are sound.
I do think that given the fragile nature of shell scripts people tend to write their operation workflows in a pretty idempotent way, though...
you can define + declare ordering with dependency specification on the edges of the graph (ie A must run before B, but B can run as often as you'd like within 10 mins of A)
Another thing is that you'll need branches. As in:
- Run `foo bar baz`
- If it succeeds, run `foo quux`,
Else run `rm -rf ./foo/bar` and rerun the previous command with `--force` option.
- `ls ./foo/bar/buur` and make certain it exists.
Writing robust runbooks is not that easy. But I love the idea of mixing the explanatory text and various types of commands together.
- in excel
- in a confluence document
- in a text file on your desktop
The use case this addresses is 'adhoc activites must be performed without being totally chaotic'.
Obviously a nice one-click/trigger based CI/CD deployment pipeline is lovely, but uh, this is the real world. There are plenty of cases where that's simply either not possible, or not worth the effort to setup.
I think this is great; if I have one suggestion it would just be integrated logging so there's an immutable shared record of what was actually done as well. I would love to be able to see that Bob started the 'recover user profile because db sync error' runbook but didn't finish running it, and exactly when that happened.
If you think it's a terrible idea, then uh, what's your suggestion?
I'm pretty tired of copy-pasting commands from confluence. I think that's, I dunno, unambiguously terrible, and depressingly common.
One time scripts that are executed in a privileged remote container also works, but at the end of that day, those script tend to be specific and have to be invoked with custom arguments, which, guess what, usually turn up as a sequence of operations in a runbook; query db for user id (copy-paste SQL) -> run script with id (copy paste to terminal) -> query db to check it worked (copy paste SQL) -> trigger notification workflow with user id if it did (login to X and click on button Y), etc.
I just think it's pretty easy to do things like start a flow back up halfway through the book and not fix some underlying ordering issues.
With scripts that you tend to have to run top to bottom you end up having to be more diligent with making sure the initial steps are still OK because on every test you tend to run everything. Notebook style environments favor running things piecemeal. Also very helpful! It introduces a much smaller problem in the process of solving the larger issue of making it easier to do this kind of work in the first place.
“A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. *All in a modern, AI-native editor.*
Why does it need to be in a “modern, AI-native editor”?
(Closing tab, flashing marimo out of brain)