Comment by SOLAR_FIELDS | Hacker News Reader

Comment by SOLAR_FIELDS | Hacker News Reader

Parent Full thread

I’ve been using Dagger for this replacement specifically. But that is ci/cd specific to caching and workflow execution. For things like workflow automation and orchestration I would reach for something like Prefect or Dagster. The point is to be able to do something in an actual programming language so that you not only get typing, readability, reusability, unit testability, local execution and language specific tooling, but also that it doesn’t suck for end users to write, debug and maintain. This also gives end users an escape hatch when your abstraction is inevitably not going to be good enough for them

Cuelang etc like siblings mentioned are decent enough but the real scalable solutions here are made available in general purpose programming languages.

the "workflows as code" is something we are thinking about too. I guess there are pros and cons for every approach and eventually we would want to support both (but need to start with something)

Check temporal.io, they are used by Uber, Netflix, Datadog...

temporal is for managing events. How is it related to ci/cd? Generally curious.

It’s more generally a workflow manager and orchestration tool. It is kind of a more general version of the tools I mentioned, Dagster and Prefect. It can be used to spawn and manage CICD tasks asynchronously.

Though I will say that Temporal's use case is probably not really well mapped to CI/CD - though it could be used for it (which is why I didn't mention it). It's primary strength is robust, long lived workflows with intelligent retries and the like - you typically want your CI/CD to be as fast as possible and while you want retries and resilience etc it's not as important as some other things (like being hermetic, reproducible, and cached).

Here's a Temporal v Prefect comparison I wrote: https://community.temporal.io/t/what-are-the-pros-and-cons-o...

tldr is Temporal is more general-purpose: for reliable programming in general, vs data pipelines. It supports many languages, and combining languages, has features like querying & signaling, and can do very high scale.

CI/CD is a common use case for Temporal—used by HashiCorp, Flightcontrol, Netflix: https://www.youtube.com/watch?v=LliBP7YMGyA

Reply on news.ycombinator.com