I keep toying with the idea of writing a cron that implements a Poisson process. Say I give it a parameter of 3600; our `pcron` would ensure the jobs occur randomly but average out to once per hour, making the timing of the next run independent of the last via the memoryless property of the exponential distribution.

The next sleep interval would be calculated probably as as t = -\lambda \ln(U) (where U is a uniform random variable). This way you ensure that the probability of the job firing in the next 10 seconds is the same whether the last job finished an hour ago or just five seconds ago. But \lambda remains the average amount of time between jobs.

It’s compelling to me because it solves thundering herd problems at the architectural level, and also because it simply seems like a lot of fun to have to code very defensively against such chaos. Switching back to a deterministic schedule after surviving such chaos probably leads to a much more robust system overall.