Added docs and an example with docusaurus: https://harshadmanglani.github.io/polaris
36 karma · joined March 4, 2024
Thanks for sharing your work Deepak, you have some pretty extensive documentation! Funnily enough, the Golang framework is almost a clone of https://github.com/flipkart-incubator/databuilderframework (another orchestration engine in Java).
This is another HN post that extensively covers almost every major orchestrator in the market: https://news.ycombinator.com/item?id=24216317
As for multi version workflows, I suppose that will have to be a tradeoff between maintaining somewhat redundant code or adding workflows as WorkflowV1 and WorkflowV2 and stitching together relevant steps in respective versions (reduces redundancy to some extent but won't eliminate)
I've noted all of these and I'll modify the README to include them. Thank you for taking the time to go through in such detail :)
Given that the current state of a workflow:
- is inherently invisible
- all we can really check in the DB is if, for a workflow, the available data contains the target data;
how would I address observability concerns?
This is a function of a lack of workflow states due to the lower levels of abstraction it operates on. User defined workflow states would do the trick, but I suppose that would take writing some more code after integrating the framework.
- with control on the database reader yourself, i think you should be able to find a way around saturation/desaturation?
- again, the fairly certain you can limit DB readers in the DataStore interface you pass to the orchestrator. I'll think about the cpu and memory share and if there's a way to expose that.
A major concern I've always had with workflow orchestrators is the versioning of workflows. Think about long running workflows (>2 days). If you change your workflow logic, what happens to the existing ones that haven't completed yet?
Everyone handles this differently, and I've been thinking about a generic way to do this. Thoughts?