Netflix has open-sourced its Maestro Workflow Orchestrator
github.com
github.com
To me seems to overengineered for most of the companies outside there, how many people do you need to manage it?
> … schedule hundreds of thousands of workflows, millions of jobs every day and operate with a strict SLO of less than 1 minute of scheduler introduced delay even when there are spikes in the traffic.
I wouldn’t necessarily call it over engineered if it can handle this type of workload.
But if you are running a fraction of these workloads with a much more relaxed SLO (service level objection), then yea this is overengineered.
Like any solution from FAANG, or “big tech”. It works for _them_ at scale but for 90% of companies the sheer management of this service will quickly outweigh the benefits.
In other words, learn it now and get it on your resume, because within a year every 200-hit-per-day service and site is going to rearchitected to use it.
It would be way too ambitious to call this thing an orchestrator, it seems to be just a primitive distributed DAG executor.
It's hard to take your comment seriously when this was one of the earliest talking points.
> Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc.
https://netflixtechblog.com/maestro-netflixs-workflow-orches...
More discussion on official post: https://news.ycombinator.com/item?id=41037745
Here's their tech blog post about it: https://netflixtechblog.com/orchestrating-data-ml-workflows-...
> Did not resolve 'com.github.vmg.protogen:protogen-annotations:1.0.0' which is part of the dependency lock stateTakes a few mins to get all the dependencies and build, but it definitely works.
Perhaps you're missing some dependency steps before hand if you're hand cranking gradle.
./run.sh maestro
This is not it however.