Most folks here are focusing on the scheduling part, but change detection means you need some way to persist state (or have a long running process that keeps state in memory).

While you could do FaaS (e.g. AWS Lambda) + another service for state (e.g. S3/RDS/EFS), that seems like overkill.

Even today I would still do it the old way: store state in a filesystem (maybe SQLite if you have more complex needs), and configure a plain old cronjob on any of the servers you have access to.

Maybe put it in docker/dockerhub to make it easier to run.

Surprisingly it is an interesting question. You have a very simple task, but ideally you want the solution to include: running environment (let's say a container), scheduler, persistent state, monitoring (you want to know when it is down). Not to mention deployment.

There are solutions for each of those, but given the simplicity it would be nice to have a lightweight solution that includes all of that with minimal configuration. I doubt it exists.

I.e. you can do all of that on AWS, but I can't help but wonder if the infrastructure setup is going to be more complicated than your actual scraper.

Edit: a good way to think about it is: imagine that a person just learned how to scrape websites, they wrote a script that works locally. How much do they need to learn to move it to the cloud?

If your answer is: ok, so you just need to create an AWS account, dockerize your script, put it in a Lambda, create RDS instance, trigger your Lambda through EventBridge and setup CloudWatch... then there's something wrong with you. There must be a better way to reliably run a fucking 10 line script in the cloud.

There's been many times over the years when I wanted to write a simple scraper that does exactly what author describes and what stopped me is the amount of infrastructure bullshit I would have to deal with.