This isn't quite that but close.
From my experience there 5 main components for larger scale system.
- Request server
- database
- queue
- background worker
- offline batch jobs
I feel like so far serverless has the first 3 fairly well developed but the last two are still underdeveloped.
But maybe I'm just not aware of other solutions?
The only issue is for background jobs you need to design it in such way to not run in to timeouts.
Which makes it slightly more complex then just having a single executable running periodically.
There is also https://www.defer.run/ which run your code on their infra and don't have timeouts. But they only support node/bun at the moment.
Currently, to get beyond the ~5 min limit per step, you'd need to deploy to something non-serverless like Fly.io, Render, or your own instance running Express.js or similar.
[1] https://github.com/bacalhau-project/bacalhau
Disclosure: I confounded Bacalhau
I took a quick look through the docs and from what I understand you can just submit docker containers to be run on a cluster. The default one is a public cluster. In some way it seems similar to Google Borg.
What I don't understand though is do you have to pay? Who provides the compute?
Maybe possible to build a vercel type service on top of the private cluster where people can just submit their code?
One tangentially related is I'm looking for somewhere to just run a python script once a day.
I'll see if I can do that with windmill.
Seems like that would be a good fit for Airplane's Schedules: https://www.airplane.dev/schedules.
It's pretty much as straightforward as deploying your Python script (and any dependencies with requirements.txt) and setting up your schedule for when you want it to run!
- workflow system