Dagger: A shell for the container age | Hacker News Reader

Dagger: A shell for the container age

dagger.io

199 pointsgk1121 comments

I feel like it's getting harder to tell what dagger is _actually_ for these days.

At first we'd hoped it could replace jenkins - it provided an alternative way to run and debug CI pipelines - right on your machine! You could write in golang and just import what you needed. The dev direction feels more scattered now, trying to replace docker, be a new shell(?), and weirdly trying to be some kind of langchain? Doing something different doesn't imply better. A new set of complicated CLI args is no better than what we started with (shell scripts, or jenkinsfiles to integrate docker builds). I'm a little bummed that the project has seemingly drifted (in my view) from the original mission.

If I may offer a different perspective: Dagger has always been a general-purpose composition engine, built on container tech. Its most successful use case is CI - specifically taking complex build and test environments, and making them more portable and reproducible. But we never claimed to replace Jenkins or any other CI platform, and we've always been very open with our community about our desire to expand the use of Dagger beyond CI. We also never claimed to replace Docker, or to "be a shell" (note that the title of this HN page doesn't reflect the title of our post in that regard).

Every feature we ship is carefully designed for consistency with the overall design. For example, Dagger Shell is built on the same Dagger Engine that we've been steadily improving for years. It's just another client. Our goal is to build a platform that feels like Lego: each new piece makes all other pieces more useful, because they all can be composed together into a consistent system.

It's a fair point - My opinions and use case are my own, I didn't mean to imply or assume there were promises not kept. The dagger team has been nothing but supportive and I do think has built a great community.

That said, in the early days it was definitely pitched for CI/CD - and this how we've implemented it.

> What is it? > Programmable: develop your CI/CD pipelines as code, in the same programming language as your application.

> Who is it for? > A developer wishing your CI pipelines were code instead of YAML

https://github.com/dagger/dagger/blob/0620b658242fdf62c872c6...

Edit: This functionality/interaction with the dagger engine still exists today, and is what we rely on. The original comment is more of an observation on the new directions the project has taken since then.

Yes it's a fair observation. In terms of use cases, we did focus exclusively on CI/CD, and only recently expanded our marketing to other use cases like AI agents. It's understandable that this expansion can be surprising, we're trying to explain it as clearly as possible, it's a work in progress.

I just wanted to clarify that in terms of product design and engineering, there is unwavering focus and continuity. Everything we build is carefully designed to fit with the rest. We are emphatically not throwing unrelated products at the wall to see what sticks.

For example, I saw your comment elsewhere about the LLM type not belonging in the core. That's a legitimate concern that we debated ourselves. In the end we think there is a good design reason to make it core; we may be wrong, but the point is that we take those kinds of design decisions seriously and we take all use cases into account when we make them.

Why not just using "Dagger is a modern compositional framework, with applications from CI/CD to AI agents", so people understand perhaps better that it's meant as a "framework" rather than a "tool" for a specific use-case?

Yes, this is what we're going for. At the moment the website says "cross-platform composition engine". But elsewhere in this thread, someone complained that it's too vague. It's hard to find a good balance between "too specific" and "too general".

I feel Nix is eating their lunch. Even though Nix as a language could be tough, it's a simpler solution than Dagger. It provides a shell, almost complete reproducibility and isolation can be added through containers. Working with dependencies to build source code using nix is usually straightforward but having the right binary version that doesn't have the same love as major languages (i.e. terraform, flux, etc) is one thing Nix really needs to implement. Marcelo's package version search[1] is a way to discover the particular hash but then you need to explicitly use that nixpkgs version for that particular binary and do a lot of mental gymnastics in your nix files. Nix could implement a syntax where you define the package versions as attribute set and then internally does a discovery of the nixpkgs hash for that versions and installs it. Flox and DevBox follow this pattern but I don't see why you need and external tool where this can be embedded in Nix (the cli).

[1] http://lazamar.github.io/download-specific-package-version-w...

IMO dagger isn't really comparable to nix.

We tried to fit dagger where we had jenkins - not just for binary builds, but for the other stuff. Mounting secrets for git clones / NPM installs, integration tests, terraform execution, build notifications and logging.

Caching is great, and dagger/nix both have interesting options here, but that's more of a bonus and not the core value prop of a build orchestrator.

Yeah, turning a lot of our Jenkins Groovy into Dagger one-liners. Makes it super easy to "run" CI locally before pushing, but you still need the orchestrator / scheduler. Also really loving the Dagger Cloud UI for build logs over Jenkins UI

We tried to fit dagger where we had jenkins

"Tried", implying it didn't go well and isn't a fit for replacing Jenkins?

I've found the Nix ecosystem to be lacking, missing packages, wrongly built packages (from the official upstream), and out of date versions. Homebrew still outclasses Nix in this regard (quality over quantity).

After taking Nix for a spin, I cannot be bothered to learn another custom tool with a bespoke language when I already have containers for doing the same things.

For Dagger, I can choose from a number of languages I already know and the Docker concepts map over nearly 1-1

> I've found the Nix ecosystem to be lacking, missing packages, wrongly built packages (from the official upstream), and out of date versions. Homebrew still outclasses Nix in this regard (quality over quantity).

That's also the case with the Docker ecosystem. On top of that, you need to take into account the base image, versions, etc.

At the end what I look for is for a project being able to build my source code with runtime dependencies and supporting tools that won't change overtime for the architecture that I need.

these days, the nix vs alts debates remind me of the emacs vs alts debates of yore

I have not yet encountered actual nix usage in my professional or open source work, so I don't see Nix as eating anyone's lunch

I happen to know Nix, but it’s from previous experience where I also learned to greatly dislike it. Anyways, did an interview yesterday where the company was using Nix to build and they were completely shocked I had Nix experience.

So yeah, no where near eating anyone’s lunch, let alone the behemoth that is Docker.

This is an interesting take, especially since I've had exactly the opposite impression. Since you mention Homebrew, I assume you're talking about running Nix on macOS, which has been eye-opening for me. The ecosystem is much larger, I very rarely don't find what I'm looking for in nixpkgs, and if it's not there, it _definitely_ won't be in homebrew. I only use homebrew for casks nowadays (and even that is managed through nix). There's also been times when I've tried to install something from homebrew on a colleague's machine, and it was a huge pain to set it up correctly (having to manually install some dependencies) whereas everything worked without effort using nix. The only downside imho is that package authors tend to go for updating homebrew first if anything, since that's considered the default. Nix will typically get updated version a few days later, but that's fine for me.

Homebrew supports both Mac and Linux

I definitely encountered packages available in homebrew but not nixpkgs, so the idea that if not in nix not in brew is wrong. Another package I use has been out of date for months, again it's quality over quantity and nix lacks the quality that i deem more important for myself

Someone else published a nixpkg for my project, but it is wrong. As an OSS maintainer, I have reduced the number of places I publish, too many packages managers these days for my limited time

> I've found the Nix ecosystem to be lacking, missing packages, wrongly built packages

This is an interesting problem you've faced, their marketing claim is that they have the most comprehensive catalog of packages (and I'm inclined to believe it). I very rarely run into broken packages, and that's usually resolved by using a stable release for that specific package - and it's not like my usage of packages is lightweight (7292 lines of nix config). That's on NixOS (and Silverblue, and Ubuntu) at least.

That's on NixOS, but on other distros there are issues. When I tried nix last year, installing alacritty, for example, required an opengl wrapper. Neovim couldn't compile plugins without environment hackery.

Things just work with pacman or dnf.

The non-NixOS OpenGL/gpu story is unfortunately tricky as it requires a binary matchup between a userspace library and the kernel driver—which on non-NixOS is unknown to the Nix package manager since it's being managed by whatever the host is.

This is arguably a flaw in how Nvidia architected this, but it is what it is. The classical solution is nixGL [1], but this requires downloading special nixified versions of the libraries on every host system kernel change. An alternative option that I was involved in the development of is nix-gl-host [2], which manipulates the LD_LIBRARY_PATH for a Nix executable to allow it to link patched versions of the host's own userspace GL libraries.

[1]: https://github.com/nix-community/nixGL

[2]: https://github.com/numtide/nix-gl-host

theoretically you could always use the previous version / iterate on them.

I think this approach of see what sticks and them trying out a lot of different things can be nice but not in its current form.

Although I have installed dagger to give it a try and I am just not sure how to make it work , the quickstart / hello world doesn't' work.

No I don't want to make an AI , I just want to see what you really are.

And so I do agree with your statement!

Yup - Ever since the introduction of llm as a core API we're basically going to stop upgrading / using the project unless things start moving in a more sane direction.

Why is that? It’s just an extra type that has no impact on any other types. It’s a tool in your toolbox that you can use when you need it and ignore otherwise.

Asked the opposite way - why is this a core API type of a buildkit interface? Having API calls to various LLMs as core functionality of a build system is just straight up weird. Doesn't fit, confuses developers on my team when they try to read/contribute to our shared build libraries, worries me about the direction the project is going.

Seems strange that especially with the push for modules this was integrated as a core type. It has nothing to do with buildkit or builds.

Dagger isn't just a build system. I use it for all sorts of things, one example would be translating Go types to CUE, or the other way around. This happens in a container.

With LLMs becoming core to the development workflow, it kinda makes sense to have a primitive, since LLM i/o is a bit different from other functions. I haven't tried it, but this thread had me go look at the details and now it's on the roadmap. Probably make something of an agentic workflow that uses a tool in a container, see how it works out. I'm still skeptical that Dagger is a good tool or ecosystem for LLM work without integrations to all the extra stuff you need around LLMs and agents.

Dagger aims to be more than a build kit interface and more than just a build system. Many people have been using it outside of CI for years.

But I can appreciate your perspective. Thank you for sharing your point of view.

This is very cool, I often find myself composing together a mixture of Dockerfiles and shell scripts for various images (build X Dockerfile for Y arch, save it as a base, build Z Dockerfile on top of the last thing, with W args). And then having to run it different ways with different mounts, based on if you're on a developer machine/robot/CI, running as a service/adhoc/attaching. This _looks_ like it could solve some of that messiness. I like that it can just reference the output of a build without having to muck around with tags (which are system global and can collide if you're not careful).

I have a particular hate on for the sandwich pattern where you have a Dockerfile to a "build" environment that then gets entered, and a bunch of stuff done in there, and then the results ADDed into a different Dockerfile for the publishable asset. I know multi-stage builds sort of address this need, but a lot of people aren't using them or have found cases where it's desirable to have the "running" context for the container, for example if a bunch of stuff is being volumed-in (which is now possible at build time but is janky).

Or there's the copy my ansible playbooks into the container and run them there as localhost approach, which can be mitigated by what Packer does with at least running the playbook over SSH. But then Packer has no concept of multi-stage or copying in assets from other containers.

There's a lot of room to do better in this space.

I don't like the sandwich either, it's too easy to accidentally miss a dependency. What I find myself doing more is building one Dockerfile as a base, and then doing `FROM $BASE` in the second to ease readability, caching, image pinning, etc.

I think it often comes about because there's a desire for developers to be able to locally reproduce what is happening on CI (good!) and having an end-to-end Dockerfile can be a great piece of that (yay!) but as soon as devs are debugging a problem or wanting to change a dependency, attach a debugger, or whatever else, it becomes too burdensome to rebuild the whole thing every time. So you end up with the setup and finalization of the build in Dockerfiles, and then some nebulous semi-manual stuff occurring in the middle.

This is workable-ish for the base developer + ship a product use cases, but it creates a bit of a crisis of repeatability when you've got a complicated CI config implementing all this mystery meat that falls in the middle of the sandwich, and there's separate documentation somewhere else that explains what the dev workflow is supposed to be, but no one is explicitly tasked with keeping the CI script and dev workflow in sync, so they inevitably drift over time.

Plus, long and complicated CI-specific scripts are a smell all on their own, as was discussed here on HN a few days ago.

Definitely. The more than can be shared between these flows the better.

> But then Packer has no concept of multi-stage or copying in assets from other containers.

I don't know about containers specifically, since I've never bothered to use packer for that process, but it does seem that packer supports multi-step artifact production and their example even uses docker to demonstrate it https://github.com/hashicorp/packer/blob/v1.9.5/website/cont...

OMG https://docs.dagger.io/features/debugging - Docker hasn't supported shelling into a broken build for a while now, this sounds so nice. I see there's some Dockerfile support, I wonder if I can just use this as a drop in replacement for `docker build` and get a nicer experience.

Yes you can.

Dagger is built on the same underlying tech as docker build (buildkit). So the compatibility bridge is not a re-implementation of Dockerfile, it's literally the official upstream implementation.

Here's an example that 1) fetches a random git repo 2) builds from its dockerfile 3) opens an interactive terminal to look inside 4) publish to a registry once you exit the terminal:

  git https://github.com/goreleaser/goreleaser |
  head |
  tree |
  docker-build |
  terminal |
  publish ttl.sh/goreleaser-example-image

> 3) opens an interactive terminal to look inside 4) publish to a registry once you exit the terminal

It seems like it would be good to be able to prevent the pipeline from publishing the image, if the inspection with 'terminal' shows there's something wrong (with e.g. 'exit 1'). I looked a little bit into the help system, and it doesn't seem that there's a way from inside the 'terminal' function to signal that the pipeline should stop. Semantics like bash's "set -e -o pipefail" might help here.

with-exec lets you specify that you want a command to succeed with e.g.

  container | from alpine | with-exec --expect SUCCESS /bin/false | publish ...

If you try that, the pipeline will stop before publishing the image.

Huh, you're right, I would expect `exit 1` in the terminal to abort the rest of the pipeline.

By the way, in your example: `--expect SUCCESS` is the default behavior of with-exec, so you can simplify your pipeline to:

  container | from alpine | with-exec /bin/false | publish ...

Thank you! Would you be willing to open an issue on our github repo? If not I will take care of it.

If that docker-build command fails, will the interactive debugger do something sensible?

Assuming you use `dagger --interactive`, the debugger will kick in so you can inspect the failed state.

In the specific case of Dockerfile compatibility, I don't actually know if it will be smart enough to drop you in the exact intermediary state that the Docker build failed in, or if it reverts atomically the whole 'docker build' operation.

You sure can :)

Dagger can read dockefiles as is, https://docs.dagger.io/cookbook#build-image-from-dockerfile

Being able to drop into failed container builds / runs has been game changing, saved me mucho time. If you are being programmatic, you can also inject the drop into interactive mode anywhere you want, without needing an error. You can use this to build up working environments into which you drop developers (kind of like dev containers)

Yeah for sure, this was specifically designed to solve the problems you describe!

I solve most of those issues with a Docker Bakefile; I'm confident I could solve the rest with Bakefiles if I had to. Reasonable developer experience.

The last time I tried something more than Buildx that Docker itself put out the experience was bad - something about not properly caching. I'll have to give this another shot sometime, though.

I had missed the fact that Dagger is apparently trying to... replace? Docker for container definitions. That's a pretty big vision.

I really like the ambition here; it's very hard for me to translate this into belief that I could start using this now and actually replace some of my existing tooling.

At the same time, I kind of hate that they went for bash-compatible. I know everybody thinks bash is unbeatable, but at some point _surely_ we're going to move past its awful syntax and footguns (and SQL, and can I have a pony?)...

In theory you could replace much of Docker's low-level tooling with Dagger. In practice, that's not what we're trying to do. There's a lot of inertia to ripping out existing tools, and even if the replacement is better, it may still not be worth the effort.

What we're focusing on is green field application of container tech. Things that should be containerized, but aren't. For example:

- Cross-platform builds

- Complex integration testing environments

- Data processing pipelines

- AI agent workflows (where you give a LLM the ability to perform tasks in a controlled environment)

In those kinds of workflows, there is no dominant tool to replace - Docker or otherwise. Everyone builds their own unique snowflake of a monolith, by gluing together a dozen tools. Dagger aims to replace that glue with a modular, composable system.

this is helpful. and i appreciate the intellectual/technological humility.

i think there's a decent chance we end up giving Dagger a spin this year.

Does it replace Kabuki in non priviledged CI builds? Can one exchange lower / independent layers like with nix container builds?

> Does it replace Kabuki in non priviledged CI builds?

I have never heard of Kabuki, and couldn't find it in a quick web search. Did you mean Kaniko?

> Can one exchange lower / independent layers like with nix container builds?

Yes.

How does that work? Are you building special adaptors that understand how package manager metadata files work? A deb package's postinstall is basically just a blob of bash running as root with access to the whole filesystem. If I tell dagger to install three packages in three different steps, those are not trivially reorderable operations. Nor are they reproducible at all, without Dagger having additional insight into the state of the local apt cache and what is offered by the remote apt repo at the moment of execution.

Yes, Dagger provides the core system API that allows creating and combining filesystem layers in any order you want. However it doesn't provide compatibility with existing package managers: that's something that you, or someone else in the Dagger community, would implement. You can extend the Dagger API with your own types and functions, then share them as reusable modules.

For example, here's a Dagger module for building Alpine Linux containers in an order-independent way. Under the hood it's derived from the 'apko' build tool created by the folks at Chainguard. https://daggerverse.dev/mod/github.com/dagger/dagger/modules...

And here's a Dagger Shell command for building a container using that module:

  github.com/dagger/dagger/modules/alpine | container --packages=git,openssh,curl | terminal

You mentioned deb packages. Your intuition is correct, Dagger doesn't magically make .deb or .rpm packages reorderable, since those packaging systems are designed in a way that makes it difficult. But it does provide the primitives, and a cross-language programming environment for creating the "adaptors" you described in a way that maximizes reuse. If someone does for deb what Chainguard did for apk, you can trivially integrate that into a Dagger module.

Neat! Okay, yes, I can definitely see the value here, particularly if all or most of an image is otherwise able to be expressed in terms of things that are deterministic, such as language ecosystem package managers with lockfiles.

It seems like you're cutting an interesting track on providing a better way than Dockerfiles with their imperative layers, but pragmatic enough to not insist on taking over the whole world like Bazel or Nix.

Yes Kaniko (my mobile keyboard did this weird autocorrect, sorry)

OK, just making sure :)

To answer your original question then: no, Dagger cannot fully replace Kaniko because it requires container execution privileges.

The Bash compatible syntax is just one representation of an underlying cross-language API. You can also write code in your preferred language, provided there's an SDK for it. (Currently: Python, TypeScript, Go, PHP, Java, Elixir, Rust, .NET)

But if you're talking specifically about shell or a "native" REPL in a safer language - I also want to scratch that itch. Some day I'll dust off https://github.com/vito/dash :)

They even supported CUE at one point. https://dagger.io/blog/ending-cue-support

I worked with the Dagger team on that. It was super hard to marry two DAG engines that work in opposite directions. The team eventually heard from users they wanted to write in the language of their choice, so there are now a bunch of SDKs.

I think one could wrap the Go SDK with CUE, and have oft considered doing it, but the work has not migrated from the backlog yet. I've been toying with a CUE powered monorepo DX CLI that is more akin to BuildPacks, using Dagger as the containerization layer.

I feel this is much more powershell like than bash. The container thing looks much more than a typed object than a plain character device/pipe. Also what is the bourne again part?

It’s def closer to powershell in all the best ways IMO. But the key point is that the container thing is actually a typed object. So is every other primitive in there including once you make yourself.

okay, I think what I ultimately missed is that this is just a new language for the existing engine. I may not be the world's most careful reader.

> That's a pretty big vision.

Is it? Docker is quite long in the tooth and this point and is a long way from perfect. Docker's design is rife with poor choices and unaddressed flaws. One I've been tracking for years: there is no straightforward way to undo several things that get inherited by derived images, such as VOLUME and EXPOSE, and Docker Inc. doesn't care no matter how long the github threads get.

So I think there is ample room for alternatives.

I see this more as a complement. I wouldn’t stop using Dockerfile for standalone definitions, but for shell+docker+shell sandwiches (the second shell being inside the container), this looks very handy.

What kind of tooling do you have in mind to replace?

Also what would you prefer to see instead of bash compatible?

i want to move away from GitHub Actions and I also want to move away from some bespoke monorepo tooling for something that's a better overall orchestrator of 'things'. Dagger scratches some of those itches, or seems to, so I've had my eye on it for a while. Knowing that they're trying to also help me define and use containerized environments - I sort of assumed I'd have to bring those myself.

I'm using Dagger in a monorepo setup (actually several). For the main one, I've gone the route of wrapping the Dagger Go SDK with a custom Go CLI (plus CUE as another centerpiece). For sure, I've been making the team aware of the challenges and rough edges. (of course there are a lot of ways people organize monorepos, we are polyglot with one go.mod but multiple package.json...)

I'm largely replacing Dockerfiles and Makefiles with a custom CLI that wraps Dagger. Being able to use a proper language, functions, packages, imports is so much better by comparison. (ymmv)

I mean if we're going full pie in the sky, fully declarative containers exist today:

https://github.com/nlewo/nix2container#nix2container

Dagger looks nifty, but it's too bad that it's still basically an imperative "start from thing, do stuff to it, publish" kind of model, rather than just expressing a desired end state and letting an underlying engine find its way there.

Dagger is fully declarative. It's just built on a dynamic declarative API, instead of a static declarative DSL.

So, if you took Nix, and replaced the static scheme-like DSL with a proper API, and then built SDKs in 5 languages for that API; and then built a bash-like shell also for easy scripting; then you would start to have something that approximates Dagger.

I stand corrected! I shall investigate more closely with this additional context. Thanks.

Thanks for giving it another chance! We need to get better at explaining all this, it's a lot to unpack, although prior experience with declarative systems like Nix or Bazel does help a lot :)

We have a very active discord, feel free to come by and ask all the tough questions!

Someone already built a web UI for composing Dagger Shell scripts in a notebook format: http://docs.runme.dev/guide/dagger

It's really neat, I recommend checking it out.

woah , that actually looks really nifty.

I am imagining this with a simple cloudflare tunnel and self hosting gitlab and I am really really seeing an open source way that developers can REALLY scale.

I mean docker is really great but dagger in notebook formats just seems really cool ngl.

love the enthusiasm. cocreator of Runme here.

the best part with the Dagger + Runme combo is that it runs entirely local. this isn't just a huge with for portability. it also cuts down development cycle times significantly.

Nice! Would be lovely to have a few "workflow examples" and tutorials which really showcase the capabilities of this setup on your website, I'm not exactly sure I understand yet in what kind of cases it really shines

Absolutely. It's in the works. I'll be sure to drop additional resources once they are available.

Resources available so far: - https://docs.runme.dev/guide/dagger - https://github.com/runmedev/vscode-runme/blob/58ea9a10c00df7...

Was curious what Dagger does and this is what the homepage says

> The cross-platform composition engine

> Build powerful software environments from modular components and simple functions. Perfect for complex builds and AI workflows.

This is generic to point of being useless.

Everything is a composition engine. Javascript is a composition engine. macOs is a composition engine.

I would lump dagger in with JS (programming language, ecosystem, runtime) and MacOS (operating system) before comparing it to any single tool.

Answering "what does dagger do" can be tough, because there are very broad applications.

Its always hard to describe general purpose platforms like these. Dagger is an open platform for building composable software. It has SDKs, sandboxed execution environment, observability, and now this shell interface.

Somewhat related, self promo

Docker shell container - https://github.com/jrz/container-shell

That looks very neat! Let me see if we can integrate it with Dagger :)

Is the objective to get inside a container to do dev stuff? Reminds me of https://www.jetify.com/devbox and https://flox.dev/

In clear terms, what would I do with this? What activity does it help me with? What program do I replace with it?

"The devops operating system" does what?

The article gives a few examples of concrete things you can do:

- Build, run, and publish containers

- Containerize language-specific builds

- Run integration tests in ephemeral environments, in a repeatable way

- Run your tools in controlled sandboxes with only the files, secrets and network access they need

It also mentions what you might replace with it:

> When a workflow is too complex to fit in a regular shell, the next available option is often a brittle monolith: not as simple as a shell script, not as robust as full-blown software. The Dagger Shell aspires to help you replace that monolith with a collection of simple modules, composed with standard interfaces.

I already build containers with Docker. I already have Makefiles that build containers and run commands in them. I already replace my complex shell scripts with Python scripts.

How does Dagger help? I just can't get anything concrete from their site.

If you’re working on your own you may not need dagger.

If you’re working on a team often your shell scripts, python programs, and makes files are not always portable between local and CI, but even worse between your local machine and your colleagues local machine. This is where dagger shines because it lets you do all that stuff in a fully portable way.

I need an example to illustrate that, because I had to read this thread to even understand what I was reading/looking at.

Maybe the docs will help? https://docs.dagger.io

A lot of projects and teams have been successfully using all tools mentioned in both CI and local dev envs. Portability and reusability of this “glue” stuff is a nice indicator of engineering maturity and culture within those projects and teams.

Low pro teams and individuals will happily abuse any new tool.

Mature teams stick to local-first CI - i.e. be able to run pipeline locally - and then translate to whatever is being used in their build and test infrastructure.

My initial impression (looking at the examples, not actual usage) is that this feels like a weird half-way between Dockerfile and using actual code to define and compose software (i.e. like Nix). Not super appealing to me as a heavy Nix user.

Did Dagger pivot their product ? I seem to recall the main selling point was an Independent pipeline-as-code service. Looks like they’re trying to rebuild Docker now?

It's the same product. Dagger Shell is a new client for the same Dagger Engine and its API. It allows you to do more from the command-line, before you have to switch to code. But the SDKs and "pipeline as code" are still there, unchanged.

It would be better to not confront potential users with a long text about unix history, nobody has time to read that. Instead just write a few bullet points about what it actually does.

However, I am very happy somebody finally put the prompt on TOP of a terminal window, since years I am wondering why people like to bow their head to the bottom of the screen.

I guess it is the expected gesture for IT devs and admins nowadays to follow and bow your head, so this is a great help for people to understand what is the right posture in life!

> We still use the shell, but it's no longer the universal composition system it was meant to be.

Huh? When did that change?

Dagger seems interesting, but this opinion that the shell and standard Unix tools are archaic is a dubious starting principle.

Every now and then a project with similar ideas comes along, and whether it rejects the notion of passing unstructured data between commands, addresses the footguns and frustrations of shell scripting, or is built with a more modern and "safe" programming language, it ultimately never seems to catch on as much as traditional Unix shells and tooling has.

The reality is that these tools and the design choices made 50 years ago are timeless in nature. They're deliberately lacking in features and don't attach themselves to any specific tech du jour. It's this primitive nature and the "do one thing well" philosophy that makes them infinitely composable. The same pipelines Unix nerds were building 50 years ago to solve problems are still useful today, which is remarkable when you consider how quickly technology moves.

Sure, new tools are invented all the time, and they might do things better than old ones. I use `eza` instead of `ls`; `fd` instead of `find` (mostly); `rg` instead of `grep` (mostly); `fzf` is a pretty essential addition to my workflow, and so on. But the underlying principles of these tools are still the same as the tools they're replacing. They're just slightly modernized and ergonomic versions of them.

Whether or not we need a `container` command, `from alpine`, or an entire new shell for that, is a separate topic. It could be argued that this could be accomplished with a few functions or standalone commands. Even if we do need this new tooling, that's great, but don't tell me that it's meant to replace a proven set of tools and workflows[1]. When containers stop being popular, will we still need this?

Also, "Daggerverse" and "modules"? Great, let's bring in the npm mentality to the shell, just what I needed.

[1]: Ah, they don't, it's meant to serve as a complement. Alright, fair enough. I'm lowering my pitchfork.

I'd like to use this tool as a breakpoint into a container. e.g.

   dagger Dockerfile L4

and it pops open a shell at that point. Then I don't need new syntax.

earthly has something like this. it's wonderful.

https://docs.earthly.dev/docs/guides/debugging

there's also https://github.com/ktock/buildg

looks promising thanks!

You can call 'dagger -i' to enable interactive debugging. If a step in your pipeline fails, you get an interactive terminal to inspect the current state.

I tried using nushell and had it as my default myshell for weeks but eventually it had too little familiarity and I settled on fish. So I’m probably not going to try this shell as my default shell. I think it’s going to be an uphill battle for them to get much adoption.

Maybe the sandboxing will be nice and they can provide it in a way that isn’t tied to a particular shell. However sandboxing on a shell command level is too coarse grained for some things.

Hi Ben! You should definitely not use Dagger Shell as your default shell. It's meant to complement it rather than replace it.

From the post:

> Dagger Shell isn’t meant to replace your system shell, but to complement it. When a workflow is too complex to fit in a regular shell, the next available option is often a brittle monolith: not as simple as a shell script, not as robust as full-blown software. The Dagger Shell aspires to help you replace that monolith with a collection of simple modules, composed with standard interfaces.

So part of the idea would be to keep my secrets out of my default shell? I’ve been looking for ways to do that but I would want specific commands to have access to them. For instance I would just want git and gh to have access to my GitHub credentials and for another program not to be able to spawn git or gh with these credentials. It would also be important not to be able to accidentally run one of these by copy and pasting something. It seems that it would need to at least be partly taken care of by my default shell for it to be usable though.

Hmm. Sounds as an interesting problem.

But how can you really differentiate b/w a user opening git and some other program running git.

I think we would need friction there , some sort of manual intervention.

The best I could think of was something like bitwarden/keepassxc like cli where it requires a password and it would just straight up copy that token into github.

If we are really talking / you have the source code and you want end to end security , you could theoeretically also compile git with the specific idea / implementation of whatever encrypted password manager you might use directly within the code of git / github but I know that can be an overkill for this problem.

What I’d want is to be able to run top level commands differently from other commands and being able to have git and gh be a wrapper that injects the permissions. They could also filter the arguments and environment variables, which I know is hard to get right. Subshells and other programs would be able to run git and gh, but not with the permissions.

I could even run git and gh in a container that has a volume to be able to access the directory.

I think I have an idea of what this could look like and I might try and prototype it with fish and see what code parts it goes down to gauge how secure it’s likely to be.

Well do let me know! sounds really interesting!

Okay, but _can_ you use it as your default shell?

It's not much of a shell otherwise, is it?

Looks weird, sorry.

Bash pipes make intuitive sense, you’re taking the output of one process and pushing it into another. Here, the use of pipes makes little sense. Why is “from alpine” being passed into “with-exec”? What is moving between those processes?

I see it as just the builder pattern that's pretty common in languages like Java. `container` is constructing the builder itself and eventually `terminal` (or other commands) use the state that has been incrementally built up in the builder to actually do something with the container.

Its the same idea. You are taking the instructions in with-exec and pushing it to the previous type.

In this case, `from alpine` is a function attached to the container type that has many additional functions. You chain them together to do stuff. You can do it through code as if it was any other object, but this shell allows you to do things without code as well.

Perhaps the example is too simple to feel useful, but being able to pipe primitives like files, directories, containers, secrets, and even any custom object makes it possible to rapidly experiment with and compose pipelines.

You're correct. As other comments pointed out, this is an OOP builder pattern:

`container().from('alpine').withExec(args: ["apk", "add"]).terminal()`

If it was unix pipes, they'd be context independent tools being composed together.

Dagger can be a useful but I feel a bit deceived, and it makes me worry about the rest of the quality of this product.

It’s similar to a builder pattern but not just a builder pattern because depending on the type you can indefinitely pipe context independent types.

So it’s a bit of a hybrid between bash and powershell.

Yeah this is method chaining, and method chaining is not the same as piping.

This reminds me of

https://github.com/progrium/envy

https://vimeo.com/131329120

A: Let's do stuff with containers.

B: What's a container?

A: It lets you execute a sequence of commands in an isolated and controlled environment that can be readily rebuilt, cloned or disposed of.

B: How do we do that? Using a standard procedural format such as a shell script or sequential command list?

A: Far too simple. Let's pipe everything through a new and evolving middleware control stack that re-implements standard primitives like file creation but lets you introduce new bugs and edge cases in any language you choose to use, thereby ensuring the host environment also requires additional configuration! And keep that interface unstable! And definitely don't write man pages!

B: TF?

Those who don't understand Unix are condemned to reinvent it, poorly. - Henry Spencer

Kinda off topic, but this page and many other pages on dagger.io seem to be loading very large GIFs that would be much better off in a compressed format (mp4 or webm).

You're right, not sure what's going on there. Escalating internally.

Right, so they've re-invented the java builder pattern, but in a shell. Why not just use a Java SDK? Or go SDK with options pattern?

Dagger does also provide a Java and Go SDK. It's all the same API underneath, so you're free to use whichever client you prefer.

Dagger is a frustrating time sink – it's interesting for what should be boring infrastructure. I've wasted too many hours debugging low-level details of buildkit and dealing with leaky kernel abstractions instead of building actual products.

Nix, Bazel, and similar alternatives may have learning curves, but they provide clearer guardrails and more predictable outcomes without exposing so many implementation details. I deeply regret the time I spent working with Dagger.

Thanks for sharing your experience, I'd love to hear more details.

Feel free to send me an email if you'd like lev@dagger.io, it would be great to learn from your experience so we can continue to improve the platform for everyone.

How come the pipe is a method call? Does this really allow composing programs?

I do all my work in containers, i'm still left asking 'why?'

So basically nix-shell ?

There’s a tiny but of overlap in principle, but you can use dagger shell without having to learn a new language and the interface is much simpler. Give it a try and let me know which you prefer!

When something breaks in dagger how do you troubleshoot it?

You have a few options:

- Dagger supports opentelemetry natively: it emits traces, metrics and logs that you can send to your favorite observability tool, or to our commercial product Dagger Cloud (which has a free tier) which lets you drill down into the trace. See https://docs.dagger.io/features/visualization

- The Dagger CLI also renders the otel traces, and you can browse them as well. Press escape, then navigate with up, down, enter, escape. Same drill-down functionality but straight from the CLI

- You can call dagger with the --interactive flag. If you do, in case of an error you will get an interactive shell to inspect the state at time of failure.

- You can also call the `Container.terminal()` function from your code, to force an interactive terminal to appear for the user at that stage. Sort of like a "debug printf" but with an interactive terminal

Reply on news.ycombinator.com