Kargo, a multi-stage application lifecycle orchestrator | Hacker News Reader

Kargo, a multi-stage application lifecycle orchestrator

akuity.io

130 pointsKenCochrane75 comments

I've used Flux, Flux2 and Argo for GitOps, and they all had their quirks. In general it seems that most CloudNative projects were kinda set in their own vision with very opinionated devs and a lot of corner cases that are not accounted for and won't be accounted for.

I have a little bit of hope that this one is going to be different with the lessons they got from Argo, but I'm not holding my breath.

Ps. when I first saw Argo, I thought this is it. The solution to all my problems.

Agreed. But I do feel Argo gives a good foundation; something on top is highly anticipated.

Akuity is getting their lunch eaten by Harness and need some kind of IP to sell besides Argo-as-a-Service. I don't really see a path forward for them in the very crowded CI/CD space, so I would wait to see how they will monetize this and who the first movers are before attempting to use it.

From a business standpoint, Akuity had their Series A in 2022 and raised $25M. They have yet to show up on anyone's radar, IMO. Maybe Kargo is their PMF but I wouldn't move my CI/CD over yet.

There is enough space for innovation for tools. Harness product is stretched thin and trying to do everything like throwing Pizzas to the wall to check if it sticks in order to boost the valuation. I doubt it will do everything well and make sense to all the users.

Some of that is a challenge of product design.

Devs tend to be opinionated on these projects because without that you end up with feature sprawl to the point that projects become unmaintainable.

On the other side, new projects need to focus on their specific segment and solve that problem well without worrying about corner cases.

Either way, any solution for a corner case needs to be implemented in a way that solves it broadly for any user.

I miss Flux (not flux2). It did everything I needed, and was simple to understand. It had its problems, but I feel like they could be resolved without a complete rewrite/revamp the Flux2 is. I understand the need for Flux2, but for my simple use-cases it's too complicated.

And Argo and now Kargo are even more complex.

I don't understand one thing about GitOps.

Imagine 10 apps deployed. All are actively deployed let's say few times a day.

You want to go back 10 days for App A. But in doing so you would have reverted whole state and all apps as they were 10 days ago.

Only way is to cherry pick particular commit and revert it.

No? I mean how git can be useful in rolling back singel components back and forth?

If you use GitOps to manage apps, you better isolate them somehow, for example put each app in a different directory. In this case, a revert for App A wouldn't cause problems for B and C.

But frankly, GitOps works best with stateless apps. Managing stateful apps is possible but you need to take care of state yourself.

You only roll forward. For live apps with evolving state this is the only safe method anyway, regardless of your deployment strategy.

You use git revert to make a revert commit (or commits, as many as you need). You don't reset to 10 days ago and redeploy that.

This one is easy. I say this in spite of the spate of different answers that say otherwise... it should be easy?

You version your apps, of course, and you would publish some artifact that represents the release. Historically this has been a Helm chart, but for Flux we are seeing many people use OCI repositories for this now. They give many of the benefits of Helm repositories without most of the drawbacks, the way that Flux uses them you retain traceability to the Git commit that started your release, and even Helm itself has already adopted OCI repositories in the current version, (just waiting for many chart publishers to catch up, we are getting there!)

The app manages its own manifests in the app repo, the app devs deploy from the main branch or a dev branch on their own app repo, but everyone else that uses the app will deploy from a release artifact. Those artifacts are tagged with semver numbers, so you can automatically move to the next version of the app as soon as its published with a valid signature.

If your app devs are the only ones using the app, then this should not change anything as they are building for production it should be versioned and managed like any production concern – whether it's for distribution or not, you still do releases.

It's not any more complicated than what you are already doing with `docker build` and `docker push` I assure you, it's nearly the same. And since those OCI manifest release tags all logically come from a git tag, there's traceability and it is still GitOps in every important sense of the word.

Automation as policy directives state declaratively that an app is always on the latest published version at any given time, a `spec.semver` with a wildcard accomplishes this very simply with a one-liner addition to your config in Flux.

When you need to roll back app A, you remove the automation (in Flux the easiest way is a wildcard) and pin your gitops config for that one app to the particular version that you wanted in git, the cluster repo, the app is pinned to the one version that doesn't have an issue. Then as the issue is resolved, you may remove the pin and put the automation back in place.

As an added benefit, you get a permanent history that shows when incident response began, how long the automation was disabled, and what version was pinned at that time, so you can calculate metrics like "MTTR" and "MBTF" that we love so much.

Personally I like to treat the Git in GitOps as an implementation detail. The underlying principle of GitOps is that your entire environments setup and config is written in code, and version controlled. So you can in theory pick any version of your entire env, throw it at blank slate, and reliably get the environment specified by that git hash.

Then there’s the whole constant reconciliation of your version controlled env specification, and the actual env, and how you automatically resolve differences. With the most important principle being that the version controlled code/config is absolute truth and something needs to figure out how to bend the world to match.

But importantly in all of this, Git isn’t that important. Version control is important, infrastructure as code is important, but Git isn’t. Arguably Git isn’t a great tool for GitOps due to issues like the ones you mention. But the huge ecosystem around Git makes the pain worth it.

I would argue the “correct” solution to your problem is a tool that automatically creates the correct cherrypicks and reverts for you based on a request to rollback application X.

Treat git as a dumb version control system, and broadly ignore “good practice”, because at lot of those good practices are designed for software development, not infrastructure development. We need to develop new working practices, built on top of Git fundamental components, rather than trying to rationalise existing working practices against the new problems that appear in GitOps.

> So you can in theory pick any version of your entire env, throw it at blank slate, and reliably get the environment specified by that git hash.

The trap here is this only works for stateless infrastructure. If you do it with stateful resources, you'll lose all data. Your gitops tool will happily recreate EC2 instances, S3 buckets and RDS instances, all empty/initialized to whatever you defined.

There are 2 different thoughts here I think. If using GitOps in Kubernetes, then application and set up (Pods) aren’t associated with Nodes (EC2). And both can be torn down and rebuilt without state issues. When state is required, then PVCs and Stateful Sets come into play.

For managed services like S3 and RDS, there are other GitOps tools like Crossplane.io which you can use for similar GitOps management. But the paradigm shift might also be that you add GitOps config to perform regular backups, and also add config to ensure that if it is being recreated, it restores from a backup.

> When state is required, then PVCs and Stateful Sets come into play.

And that's the problematic part. With GitOps, you manage the infrastructure, so in the case of PVCs these are PVC manifests - you need to manage data separately. But even if you exclude the data, even some PVC manifest changes like size change can be tricky. Also some properties are immutable (like its storage class, access modes etc.) so that you cannot modify a PVC without recreating it.

You can decide you want to avoid the problem completely and store your data outside of Kubernetes. Now you have two problems...

Thank you for the insights. Last part specifically. I also feel something is surely amiss.

To restore app A to a given state, you add git commits that specify the desired state of app A.

Basically, much of git's time traveling functionality is not of much use.

I think that's the nature of any CI/CD tool out there. All of them have some opinion to bound the product otherwise it is very hard to tame usecases.

I've only skimmed the quickstart docs https://kargo.akuity.io/quickstart/ but this snippet immediately clicks for me as something that I'd want to try

    cat <<EOF | kubectl apply -f -
    apiVersion: kargo.akuity.io/v1alpha1
    kind: Promotion
    metadata:
      name: prod-to-${FREIGHT_ID}
      namespace: kargo-demo
    spec:
      stage: prod
      freight: ${FREIGHT_ID}
    EOF

Have enjoyed using argo in general in the past, its got a great model for k8s native workflows / events but never got to using it for CD.

Yeah we use argo workflows to power some e2e testing and rollouts for canary blue/green, and their notification server, but don't actually use their cd product.

I used Argo for some e2e test suites I wrote and it was a god send. Being able to trigger off of GitHub labels was so insanely powerful for building developer tooling.

I can almost follow this but I am at the edge of where I see how it could sound like buzzword gibberish.

I dont blame the author(s) but things are getting more specialized and more terms are created. (which usually maps to an existing term which also maps to an existed term which also maps an existing term and so on.

I have though about trying to create a "terminator"(hah) where you can paste in something from a new product and it will map the terms to existing terms as a sort of translator or term machine.

"" We are thrilled to announce Kargo, a multi-stage application lifecycle orchestrator for continuously delivering and promoting changes through environments. Kargo, brought to you by the creators of the Argo Project, is a reimagining of CD pipelines for the cloud-native era, with first-class GitOps support, progressive delivery features, and is 100% open source. ""

Weird thing is, when I read that quote it makes perfect sense.

But I understand how it could look like the undecoded bytefall in the Matrix for those outside the know.

multi-stage: in this case they mean dev->test->prod

application lifecycle: CI & CD

orchestrator: a thing that runs jobs with dependencies

continuously delivering: CD

promoting changes: if the tests work in the staging environment, allow someone to click a button that says "deploy to prod"

environments: dev, test, staging, production

CD pipelines: a bunch of continuous delivery jobs

cloud-native era: microservices/SaaS/PaaS/IaaS/IaC/II/containerization/webhooks/OIDC

first-class GitOps support: if you push a commit, a job is run

progressive delivery: deploy to 10% of users, if lots of errors, roll back the deploy

100% open source: our code is [currently] available but we will charge you out the ass to manage it for you and Enterprise features will be locked up once we write them

good ones.

I think. orchestrator: a thing that runs jobs with dependencies "make"

Every time there's a new tool, it's worth considering if it locks in previous choices. E.g. ArgoCD means I can't move off Kubernetes. Will Kargo mean I can't move off ArgoCD or Kubernetes?

Using Kubernetes locks you into kubernetes. ArgoCD doesn't lock you into Kubernetes because its just a wrapper around managing kubernetes manifests

Its like arguing that cloudformation locks you into aws

I feel like Im missing something obvious but… cloudformation does lock you into aws, doesn’t it?

It totally does. And it's an interesting argument when things like Terraform exist.

Migrating from AWS to othercloud isn't really going to be about converting your CloudFormation stacks to something new (TF or otherwise), because that's the least of your concerns.

If you had Terraform-defined infrastructure for AWS you'll still need to define entirely different infrastructure for othercloud. Starting with Terraform is going to be marginally easier because maybe you intelligently organized your code using the tools Terraform offers (didn't you?), and didn't just materialize it all as a big pile of root modules (or worse, root modules instantiated by running with different var arguments--you didn't do that, did you?). Because with Terraform you can do some code reuse, etc. in a way you can't with CloudFormation.

But it's only going to be easier because you already know the language and because you have less code (maybe) to examine to inspire your new code base[1]. All your resources will be different and will mostly behave differently as well, so it's not a matter of renaming but of rearchitecture anyway.

----- [1] Well, the third thing that's going to be easier is that you'll understand the DIY aspects of actually running your Terraform, since it's not a service like CloudFormation. Wrangling states and so forth.

That's the point. In other words, ArgoCD is a tool for Kubernetes, just like CF is a tool for AWS.

Environment promotion is an area that has been severely lacking with GitOps tools. I've hacked together CI pipelines to do environment promotion with Argo CD before and it was far from my preferred approach, but I could never find a good GitOps way of handling it. Might check out the webinar tomorrow to see if this is something worth trying out.

What about having a git repo that has a Kustomize base which has overlays for different environments?

This way each environment is in its own directory which can have its own patches such as using a private load balancer instead of public for a staging environment or setting whatever environment variables that need to be different.

Then at the Argo CD level, Argo CD running in prod will look for an Application in the prod/ directory and Argo CD running in staging will look for an Application in the staging/ directory.

All in all you end up deploying the same artifact across environments and all of this is managed and exposed by Argo CD. For example you might want AutoSync enabled for staging so developers can auto-deploy their code but prod requires a tech lead or manager to push the sync button to deploy it.

The above works well in practice and isn't too complicated to pull off.

This approach definitely works, but its not the most convenient imo. At least in my experience, this always resulted in separate commits to dev/stage/prod as you progress through each part of the promotion process. It works, sure, but depending on your exact situation it can get cumbersome pretty quickly (for example, start adding in multiple prod regions, or maybe you want to canary one prod region first).

With Kargo, it looks like it lets you define your preferred promotion process, and then lets you release each promotion with a single click. I think the part that is the most interesting to me though is that it writes the changes it is making directly to the git repo.

I am a lead on Kargo. That's a great approach you've described and Kargo helps you make that approach work even better. It's not trying to re-solve problems that are already solved well by config management tools or by Argo CD. It provides a higher layer that integrates with those to orchestrate the progression of changes from one environment to another...

So in your example, suppose you changed something in your Kustomize `base/` directory. With _just_ Kustomize and Argo CD, those changes roll out to all environments all at once. With Kargo, you can progress the change and (re-)validate it environment-by-environment.

> With _just_ Kustomize and Argo CD, those changes roll out to all environments all at once. With Kargo, you can progress the change and (re-)validate it environment-by-environment.

In prod it wouldn't get rolled out until someone manually clicks the sync button for each app in that environment. But yes, in staging it would get AutoSync'd to all of the apps. I get what you're saying though.

It will be curious to see how Kargo works across multiple clusters in the staging vs prod example where both clusters each have their own Argo CD instance running but you want to view and promote an artifact across N clusters.

In any case, I'll give it a look when it's available to see how it can improve my current workflows. Overall I'm really happy I started using Argo CD ~2 years ago.

> In prod it wouldn't get rolled out until someone manually clicks the sync button

That's a great point and if that works for you, awesome! Note though that relies on the state of the prod Application to temporarily contradict the source of truth that is your repo. If you lose that prod Application and re-create it, prod will be in a state you hadn't yet blessed.

Indeed. That use case hasn't been something I encountered yet since we generally operate at a scale where our apps at the Argo CD level rarely change once they've been created.

But thanks for pointing out the mismatch on the source of truth. Definitely something to consider.

We faced that (and are facing it today) with Helm chart updates.

In our current setup it's not possible to rollout a new version of an Helm chart in dev, then in staging and finally in prod (without causing an out of sync state.

The rest is managed via values-$env.yaml, which works perfectly fine

I mistakenly read that as "Kargo, a multi-stage .... lifestyle orchestrator" and thought, damn, that sounds interesting!

I'm not a CI/CD specialist, so help me out. Is this something that is more suitable for large, complex setups or is it something smaller shops should be considering too? How does it compare with competitors like Spinnaker?

Is this tool similar to https://dagger.io/?

I think they are similar but a little different. From Dagger's website "Dagger is a programmable CI/CD engine that runs your pipelines in containers" Kargo is a tool that works with your existing CI and CD (Argo CD) to help promote your code from one stage (Dev, UAT, QA, Prod, etc) to the next.

Kargo has native support for Argo CD today, but one of it's goals is to be agnostic and could probably work with Dagger or other tools in the future.

I'm looking forward to seeing support for other GitOps tooling. Argo CD makes sense initially since this is by the same creators, but it would be nice to see agnostic tooling that can support other GitOps tools like Flux. I hope this is something that we will actually get to see and isn't just an empty promise.

Cool to see you around :)

Hey Nick, it has been to long, I hope you are doing well. :)

Not much overlap. Dagger wasn't on my radar until now, but it reminds me of something else I used to work on and it frankly looks awesome!

But it looks to allow you to defines pipelines using your preferred imperative programming language -- which is awesome. I would totally use that for CI.

Kargo is really about divorcing CI and CD and creating something that's actually CD-focused and compatible with established GitOps principles.

Sometimes, CD depends on CI. I trigger CD only when CI succeeds. Kargo looks confusing, but I have to dig deeper

Wow, looks interesting.

You make per project makefile-like python pipelines that are executed by an installed runtime/docker container?

Then I will give it a try, I've been looking for a lightweight local ad-hoc jenkins substitute.

I swear to god I thought this was dagster and was confused for a bit.

Dagger looks more comparable to Argo Workflows: https://argoproj.github.io/argo-workflows/ That's the first of the Argo projects, which can run multi-step workflows within containers on Kubernetes.

For what it's worth, my colleagues and I have had great luck with Argo Workflows and wrote up a blog post about some of its advantages a few years ago: https://www.interline.io/blog/scaling-openstreetmap-data-wor...

> Unlike CI, Kargo deployment pipelines are not generic “jobs” (...). Instead, Kargo Stages are used to model your environments (...)

I didn't see an explanation of how Stages are different than jobs. Every single usage of Stage could have been replaced with job and the meaning would stay the same.

The data and DevOps marketing people really need to drop the buzzwordism.

I am a fan of the term- after long naming wars internally we settled on promotion stage to signify the same concept. It is one or more targeted clusters that function as one of several environments. It works both in the sense of stage as venue and stage as phase. Teams promote their config from something like dev to QA to Preprod to Prod. Often there is a need for business logic to happen before advancing to another stage, or multiple environments in different regions that need to be healthy within a stage before promotion can happen.

Job doesn’t really encapsulate this at all. An alternative may be Metaenvironments.

Stages are just quality gates. When the quality level is high enough, the gate is released and the next step in the pipeline is performed.

It sounds like the solution all of us knew Argo needed to solve but didn't. I'm... hopeful? But besides the fact that it's not finished yet, it also sounds like it's going to be tightly integrated into their "Akuity Platform". Tightly integrated solutions are an anti-pattern; we'll end up with cobbled-together scripts again when something we need to do isn't allowed by the "opinionated" platform.

I had the good fortune to work with Akuity last year. I haven’t looked at Kargo yet but I’m sure it’s going to be solid based on my experience with their engineering and leadership. The problem it is trying to solve for is very real and historically expensive to solve for. Most companies that operate in multiple clusters and want to do GitOps will face this issue, especially when there are gates and approvals that happen between environments.

If Kargo requires R/W access to GitHub, and auto updates charts/images, isn’t that asking for your production environment to be infected by a change prepared and cultured in your dev environment and then auto updating / hiding itself into prod freight?

We disallow writing back to GitHub to avoid this issue, and manage stages through branches, combined with directories for overlays. Things can get out of sync, but comparing branches is easily automated.

Couldn’t this be mitigated against by using something like codeowners and only allowing access to manage versioning files?

This looks interesting, codifying and automating promotion looks cool.

In my experience it gets pretty hairy to build automated conditional promotion in a GitOps pipeline, especially if you don’t go all the way to continuous delivery and might have different versions at different stages between environments.

For those who are curious here is the Github repo https://github.com/akuity/kargo

Sounds promising. Vaguely terraform like in its application to outside subscription services.

Can’t say I’m a fan of “we call it freight” though. Artefact is a perfectly fine word.

I tend to agree, the abstraction from common language (e.g. artifact) is fun because it stays on theme with the "kargo" name but can make it difficult for new users to understand what is happening. I've always loved that in Argo CD, it's just an Application. No special name for something users are often going to refer to.

Freight often contains multiple artifacts. Container would have been a nice word, but it's taken. ;)

Kargo (cargo) and Freight makes sense no?

As of writing this, there are 5 total comments, 4 of which are by very inactive or brand new accounts with no activity, adding unhelpful/low-effort posts like:

> Something like this is definitely needed in the GitOps space...always felt like something was missing between promoting things and rolling them out

> Interesting. Will share with my team and try it out.

> There is a webinar tomorrow with Kelsey Hightower! Here is the link if you want to join https:...

> Looks promising, I'm definitely going to take it for a spin!

Something tells me this isn't organically gaining traction...

If you suspect abuse, email the mods since this sort of meta tends to go to thread hell.

Another project announcement, another situation where I would literally pay a coffee's worth to cut to the chase -- why is there a need for another GitOps tool when several already exist?

Tighter integration with other Argo products? Why is this not simple a new component of Argo CI/CD? How much is "thought-leadership" a part of it, it is Kubernetes-adjacent, after all.

So far even the answers in this thread leaving me asking these questions even more strongly? If Argo CD is a "continuous deployment" tool, why isn't the priority making staged-rollouts a first class feature in Argo CD itself?

This is coming from a place of curiosity, not simply being a miser, I promise.

There's fluxcd and Argo. What other Gitops tools one should keep an eye on?

https://github.com/weaveworks/awesome-gitops but also, like, a shell script?

I guess, gitops + gating/staging is compelling, but again, I don't see how this is a distinct product.

Depending who you ask, CD stands for one of two things: Continuous _Deployment_ or Continuous _Delivery_ and they're not actually the same thing.

Want to go camping? You go on Amazon and order a tent. An Amazon truck _delivers_ it to you. Not the same as deployment. Deployment is when you pitch the tent.

Argo CD and Flux are great at Deployment. "Make this run in my cluster."

Neither of them addresses the other need -- delivery -- getting artifacts from point A to point B with adequate quality gates, etc. along the way.

Isn’t that Argo Rollouts?

That's also deployment. It's a drop-in replacement for the Kubernetes Deployment resource type. It doesn't do anything to help you get things from point A to point B.

I don't know what this means, to be honest. Everything I build ends up as a cached Nix derivation output. Delivery is a glorified http get w/ a pre-configured public key. Beyond trivial, and even more so now that the containerd-nix-snapshotter [1] is available. (This is where my shade at the k8s ecosystem comes in, there are millions of dollars in VC funds invested in Golang projects that are obliviated by just... using Nix).

Though, I guess maybe I just need Kargo and not the rest of Argo? Can someone confirm that? I'm here and bothering commenting because I do still see the value of "pipelines" and gating. Just, incredibly skeptical of new K8s-related products using k8sy buzzwords. And this announcement blogpost is dripping in them.

[1]: https://discourse.nixos.org/t/nix-snapshotter-native-underst... (aka, just deploy nix paths as native containers and you get the power of the nix + nix-store)

Something like this is definitely needed in the GitOps space...always felt like something was missing between promoting things and rolling them out

Reply on news.ycombinator.com