Hacker News Reader

Comment by wmichelin | Hacker News Reader

How does it handle state and restarts? What happens if a job is scheduled to run "before 10am", then the entire server restarts at 9:55am, will it try to run that same job again when it boots back up?

View on HN

It will run it if the scheduler is called before 10am according to the docs - at runtime the conditions must be met.

Right but what if it already ran? Should the jobs be written such that they are tolerant to re-runs?

I think that regardless of the scheduling tool, it is very worthwhile putting in the effort to make your tasks idempotent. I've had tons of cases when I had to repeatedly rerun failed tasks "in anger", and was always grateful to know I made it safe to do so.

If it took less than 5 minutes to boot then I can't see why it wouldn't work.

How would that work for other schedulers? Also, if a server reboots that's quite bad all round anyway. Hopefully you'd be notified directly.

Other schedulers have a durable database of attempted runs. This doesn't seem to have anything like that.

Would you have a recommendation for an easy to use Python scheduler with such a feature for a personal project?

Seconding this. I would love a middle tier job scheduler / manager for Python that has persistence. I feel like there's a missing middle between cron+scripts and the enterprise grade tooling built for ETL tasks.

APScheduler

It looks pretty good. Thanks!

I think you can create ephemeral timers with Systemd if you're on Linux.

Celery Beat is pretty good

https://www.prefect.io/ is also pretty easy.

From its docs, I understand that it has a task status logging layer[1] which can persist to SQL/Mongo (using another framework called Red Bird[2]). I haven't tried them out though.

[1]: https://red-engine.readthedocs.io/en/stable/tutorial/basic.h...

[2]: https://red-bird.readthedocs.io/en/latest/

I think on the contrary, it's important to build software that's resilient to random reboots: you can't control things like losing power or your CPU failing, so the less toil you have to manage around that the better.

Reply on news.ycombinator.com