Streamlining CI/CD Pipelines with Code

The benefit of Dagger is that your builders are fully containerized, you can run CI locally, and you get to do all of that using your favorite language.

You could achieve those goals independently using things like Make + Docker + keeping CI dumb, but that is a lot of plumbing and still has edge cases. For example, Makefiles often use utilities like sed, curl, etc., which behave differently on macOS versus Linux. It also requires a lot of discipline to keep CI from growing, and you need to learn the Make syntax, which, while useful, can be a bit of an investment, especially as you get to the more advanced features/syntax.

This shines when you can write your pipelines in the same language as your app. Often, teams might have only 1-2 people who _really_ understand Bash/Make, but the rest of the team understands the language they use daily (Go, TypeScript, etc.)

Dagger lets you leverage your knowledge of more common languages to write your CI pipeline, and additionally, these builds are containerized and can all be run locally.

I have used Dagger a couple of times and liked it. One competitor in the space is Earthly [0], which has a Makefile/Dockerfile syntax. I've had good experiences with Earthly, but Dagger is a better pick for medium/large projects.

[0]: https://earthly.dev/

> Are there developers who wish this? Specifically developers who will not use it as an excuse to make an overly complex difficult to maintain solution instead? Not to mention that YAML pipelines can be maintained by a wider array of people some of who don't necesairly have the code skills to do the the same in code.

As someone who has been building automatic deployment systems for 5+ years, I do.

I'd like to create libraries and workflows that expose infrastructure and deployment best practices to developers so they can spend their time solving business issues.

Dockerfiles + Docker Compose + YAML template files are good enough for simple cases, but they will quickly break once you need to support different, possible configurable use cases.

I've been using AWS CDK in Python to expose infrastructure libraries for developers and the difference is night and day vs using plain YAML. People just build stuff and deploy apps without talking to our infra team at all, object / functions serve as neat, statically checked, documented interfaces for our devs.

I'm not sure Dagger is THE solution I'm looking for CI/CD, but having something that exposes rich declarative interfaces for automatic deployments is most definitely not a baseless idea.

To add to what's already been said: If you think about it, CI pipelines are typically a complete description of how your system is built, tested, and deployed.

Which is pretty fantastic except for how walled off they are. You can't really re-use these descriptions for e.g. development, they're not vendor agnostic, and they only way to run them is by pushing your code.

Maybe it's a silly analogy but it's almost like being a web dev that doesn't have a browser and needs to send their code to a friend who can tell them if that font size looks good.

I think we're way over due for freeing these "blueprints" of our system from the confines of CI and making them portable and flexible. And containers are the technology that's enabling that.

Full disclaimer (as always): I work at Garden[0] where we're also solving that problem but taking a slightly different approach to Dagger (it's still a DAG). Garden config is declarative and the jobs (we call them actions) have a semantic meaning. You can e.g. have a Build action of type container or a Deploy action of type Helm and Garden will figure out what to do with it.

We've also put a lot of emphasis on the inner loop development story with hot reloading functionality, log streaming and more.

[0] https://github.com/garden-io/garden

YAML is the worst of both worlds. It pretends to be just simple configuration, but it by necessity cannot be that, because it gets interpreted by the CI system. The result is an unholy language selling itself as simple JSON with comments and less quote escaping. (At least nobody got the bright idea to put jinja in there.)

The CDK people saw this and went the much saner strongly typed typescript JSON synthesis. It’s still shit but at least you get your config statically analyzed before pushing it 17 times until it finally works (because why would you want to test locally, nobody does that).

Disclaimer: I build and maintain'ish the rust sdk of dagger.

If you've ever worked with a sufficiently large pipeline, or set of pipelines, you face the fact that stuff breaks and you have to spam commits to try to get the weird yaml dsl to do the thing you want. Not that dagger is that much different in that context. But you can actually test your pipeline locally, shortening the feedback loop from 10 minutes to 10 seconds.

We've even gone as far as integrating it into our developer tools, with caching we still have nice to use clis and apps. We just don't have to think too much about deployment as we're heavily invested in docker already.

From a platform engineering perspective. We'd like actual guarantees about our pipelines when we roll them out to thousands of repositories. With dagger, we can do traditional unit tests on parts of our pipeline, and full integration test (or whatever you'd like to call them) on whole example pipelines.

This is something that would suck to do, with github actions, bash and docker by itself. And as such isn't done often, and when done brittle. Been there done that.

Another example is doing programmatic builds in a more comfortable language than Docker or bash. Lets say I'd like to solve rusts build/caching issues. With dagger, or other programmatic build tools. I can now mix docker commands, with computation to solve my problem. I can for example rely on golangs hashing libraries etc. rather than installing a bunch of packages, I don't want to end up in my final images anyway.

Dagger shouldn't be used by developers directly. IMO the same as traditional pipelines, if the developers have to touch these files, then something is wrong. Instead it should be built and maintained by a platform team, with contributions by developers whom need specific features (which dagger allows, because you can just use a regular package manager).

This may sound a little fanboyish, but we've just felt the need for less friction in traditional CI systems on actually having modern software engineering tools and methodologies available. Something which Dagger enables for us.

Dagger looks far more confusing and weird than the docker compose example.

I haven't used Dagger, but I'm familiar with the approach because Earthly has a similar approach.

Getting your CI/CD code out of some CI-only runnable format and into something you can run on your local or wherever is game-changing.

You should do it :)

Dagger fits very nice with small and large dev and devops teams. Similarly to other tools, the more you grow, the better you'll have to organize your projects to make the best use of it.

Test fakes are still a kind of white-box testing. They are built right into the code of the application and re-use the types that you define within. The kind of integration tests defined in the blog post are, in my opinion, are the most black-box kind of approach you can have. And as such, they are a great way of testing APIs that have many external clients since you are validating it as if you were one. You only look at HTTP, you don't look at how it is implemented.

> frankly not the easiest to follow I would love to know more about this. As you can see in my blog, I'm not a very frequent writer, if you have concrete feedback I can use to write better and make blog posts that are easier to follow that would help me tremendously.

> LLM? No. I wrote every word myself, even the mistakes. I based it on something I built for the company I'm currently working at.