This looks like a fantastically clean API for Python data and ML pipelines. Congratulations!
It would be great to have a scheduler and monitoring UI that are equally lightweight.
It would be great to have a scheduler and monitoring UI that are equally lightweight.
We could provide similar support e.g. for Airflow, if there's interest.
For monitoring, we have relied on notebooks this far. Their flexibility and customizability is unbeatable. We might build some lightweight discovery / debugging UI later, if there's demand. We are a bit on the fence about it internally.
This isn't a scheduler that will trigger 'flows' though is it? If I search "Scheduler" in your docs the top result is a roadmap item, and searching "Cron" turns up nothing.
Is it true that right now that to run a DAG "every day at 3am UTC" requires an external service?
This is pretty interesting approach. Notebooks are a natural environment of Data Scientist and for monitoring they'd provide really good discoverability, flexibility, and interactivity.
Do you use more traditional monitoring that will alert you somehow if a workflow fails, or a workflow hasn't run at all for X hrs/days?