BTW, here's https://github.com/pditommaso/awesome-pipeline a list of almost 200 pipeline toolkits.
* Source control for workflows/DAGs (using git-sync)
* Tracking/retries with SLAs
* Jobs run in Kubernetes
* Web UI for management
* Fully open source
and temporal: https://github.com/temporalio/temporal
I can't find an easy way to explain everything it does, but it pretty much allows you to write naive functions with no error handling, with month-long sleeps, auto-retries on unreliable function calls, etc etc.
It also gives you a web interface when you can inspect the running functions, and allows for external code (and other workflow functions) to signal/query the running workflows.
This is more of a proof of concept but it can also support DSLs (although we found the go client is easier to understand than DSLs): https://github.com/checkr/states-language-cadence
BTW. You can AMA about the technology here.
Is this "curated"? It seems like an exhaustive "dump" of toolkits.
So for example, if you said "I want to start a library, donate any books you have at my house" that would not be a "curated" collection in my opionion. If you went through & evaluated the books, selecting only those that you'd personally recommend and discarding the rest, that would be a "curated" collection.
(Strictly speaking any maintenance of a collection even just "dump everything at my door at I'll put it on the pile" could be considered "curation" but to call a list of tools "curated" suggests there's some selection going on, and there does not appear to be on this list.)
EDIT: In fact, all of the definitions of "curate" here[0] start with the word "select," for example "Select, organize, and look after the items in (a collection or exhibition)." This fits my meaning. This definition[1] of "curator" requires one to merely have "care and superintendence of something," so in that sense the list is "curated" insofar as someone looks at PRs and clicks the "merge" button.
The Miriam-Webster definition, however, speaks of curating a more general "something" rather than a "collection." If you "curate" a statue by cleaning & protecting it, fine, you don't need to select the statue. However, when a collection is curated, in my opinion, this necessarily implies selection, not just maintenance.
0: https://www.lexico.com/en/definition/curate 1: https://www.merriam-webster.com/dictionary/curator
https://softwareengineeringdaily.com/2020/04/29/prefect-data...