Ask HN: What is the cheapest, easiest way to host a cronjob in 2022? | Hacker News Reader

Ask HN: What is the cheapest, easiest way to host a cronjob in 2022?

361 pointsheywhatupboys325 comments

I thought this could start a good debate on the subject. Myself I have had to make a short running web-scraping job that, given a change in the site, sends a notifying email. This running once an hour.

It is 2022, so I had thought it would be tremendously easy and cheap, but it seems no solution is easily implemented.

View on HN

You should check out Modal! Does much more than just cronjobs, but super easy and cheap: https://modal.com/docs/guide/cron. They have an example with a simple web scraper as well: https://modal.com/docs/guide/web-scraper

From their pricing example:

> You schedule a job to run once an hour for 15 seconds. This job uses 50% of one CPU core and 256 MiB of memory. This job will cost $0.016/day for the CPU, and $0.001/day for the memory, adding up to $0.017/day, or $0.51/month.

Ooh and apparently everyone gets $30/mo of credits as a baseline, so this is within the free tier (if you don't want to pay 50 cents per month...)

This is very cool. Love the pricing model.

We use Modal at my startup, it’s fantastic.

I can't get their site to load.

Wow

Most folks here are focusing on the scheduling part, but change detection means you need some way to persist state (or have a long running process that keeps state in memory).

While you could do FaaS (e.g. AWS Lambda) + another service for state (e.g. S3/RDS/EFS), that seems like overkill.

Even today I would still do it the old way: store state in a filesystem (maybe SQLite if you have more complex needs), and configure a plain old cronjob on any of the servers you have access to.

Maybe put it in docker/dockerhub to make it easier to run.

Surprisingly it is an interesting question. You have a very simple task, but ideally you want the solution to include: running environment (let's say a container), scheduler, persistent state, monitoring (you want to know when it is down). Not to mention deployment.

There are solutions for each of those, but given the simplicity it would be nice to have a lightweight solution that includes all of that with minimal configuration. I doubt it exists.

I.e. you can do all of that on AWS, but I can't help but wonder if the infrastructure setup is going to be more complicated than your actual scraper.

Edit: a good way to think about it is: imagine that a person just learned how to scrape websites, they wrote a script that works locally. How much do they need to learn to move it to the cloud?

If your answer is: ok, so you just need to create an AWS account, dockerize your script, put it in a Lambda, create RDS instance, trigger your Lambda through EventBridge and setup CloudWatch... then there's something wrong with you. There must be a better way to reliably run a fucking 10 line script in the cloud.

There's been many times over the years when I wanted to write a simple scraper that does exactly what author describes and what stopped me is the amount of infrastructure bullshit I would have to deal with.

I thought about AWS MQTT "IoT" as the pricing contains gems like a million MQTT packets for a dollar. You'd use a MQTT "retained message", push the current SHA1 hash of the page or whatever will fit (maybe only a CRC32 LOL), then connect an hour later to receive that message containing the hash. The cost would seem to be $1 per 5 years because AFAIK AWS meters on input and output... I'm not bored enough to audit the AWS pricing structure to see if this will work. I think $2 per decade would be hard to beat for a billed service.

I think the cheapest "cheaty" way to store data for free would be to pollute tracking sites. Open an account at some company, change your mailing address every hour, etc.

Hash isn’t nearly enough. Web pages love to generate random ids, cache busters, etc.

> If your answer is: ok, so you just need to create an AWS account, dockerize your script, put it in a Lambda which is hooked up to RDS, trigger lambda through EventBridge and connect it to CloudWatch... then there's something wrong with you. There must be a better way to reliably run a fucking 10 line script in the cloud.

I mean, you don’t have to dockerize it, and I would use S3 or Dynamo for state, but otherwise it’s a pretty simple solution and it can run entirely in the free tier on AWS. It’s way better than managing your own host, especially when you consider that you get monitoring, alerting, logging, etc for free. EventBridge is just the standard AWS thing for triggering stuff (in this case, a cron schedule)—don’t let the branding scare you.

I do a lot of serverless development and agree with you on this. This is a very simple and cost effective problem for lambda and EventBridge.

I don't have much experience with AWS Glue, but would this fit the use case here?

https://docs.aws.amazon.com/glue/latest/dg/monitor-data-ware...

"you need some way to persist state (or have a long running process that keeps state in memory" Hash the data, update the script with the hash.

Put the 10 lines in a Lambda and invoke it from a schedule in EventBridge. Done. If you follow the default settings you'll get Cloudwatch logging as well.

Fly.io's Machines support cron jobs. You just hand them a Dockerfile and a schedule, and you'll be within their free tier.

https://fly.io/docs/machines/working-with-machines/#create-a...

See the `schedule` parameter.

I came here to say this!

A maximum frequency of once per hour unfortunately.

I feel like maybe this problem was "solution first". A cronjob is a solution. For example, Supabase supports cronjobs, but it can't be used to take screenshots.

This might be a non-answer, but if you just want "scheduled notifications of a website change", then you can use something completely purpose built for that. There is a Github action that does exactly this: https://github.com/yasoob/github-action-scraper-tutorial/blo...

This might also be a non-answer, but I've found myself using Integromat/Make or Zapier more and more for this type of work. It allows me to whip things up quickly and it can actually be approachable by non-technical people.

As a supabase enthusiast I take this as a direct challenge.

I like it too and I think you might be able to accomplish this with that extension that allows HTTP requests in your postgres statements but I haven't looked into it

Haha I actually tinkered with it a bit.

One thing I love about supabase is that they are focused. They have a ton of features, but they're cohesive in ways no one else has managed. You can manage your document storage, their built-in auth provider, and websocket updates ALL IN SQL.

And now their QETL and remote servers?? (https://supabase.com/blog/postgres-foreign-data-wrappers-rus...) And their CDN and Image processing???

It is shocking and almost painful how beautiful this all is. And how with those basics, you could almost do anything. But yeah, I think the only thing between Supabase and Web Scraping Screenshots is a Postgres Plugin, entirely independent of what Supabase provides.

If I were one of those beautiful people who write a blog about rasterizing on a e-pregnancy tests, I'd give it a shot haha

You've also got changedetection.io ->(https://lemonade.changedetection.io/start), which can be selfhosted on a pi etc. I've used it myself to snipe the cheapest kimsufi server.

I use Serverless with Lambda. It's even a template in Serverless, so with the CLI you just type `serverless` and select the Node.js/Python Scheduled Task template and everything is set up for you.

I do a lot of web scraping, and Lambdas are very useful for web scraping since their IPs can change for each request.

For deployment, you can just do `serverless deploy` or set up a GitHub action each time you push to main.

If you are comfortable using AWS , You could us AWS Batch (free) with Spot Fargate jobs with Cloudwatch cron as the trigger . You only pay for the vcpu hours your job consumes https://aws.amazon.com/fargate/pricing/ , it’s a set it up once and forget it model.

You could use lambda if your runtime is expected to always be less than 15 minutes Batch fargate jobs documentation - https://docs.aws.amazon.com/batch/latest/userguide/fargate.h... Cloud watch cron documentation

https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/S...

https://aws.amazon.com/about-aws/whats-new/2018/03/aws-batch...

Everything can be deployed as a simple cdk app once you have your code ready in a docker repository somewhere . There are no servers to maintain and no ongoing maintenance will be necessary

AWS’s Chalice framework also offers a high-level way to declare these scheduled functions (amongst other things): https://aws.github.io/chalice/

Haven't used Chalice, I've heard good things.

I use Zappa and it's great.

Github actions FTW! > https://docs.github.com/en/actions/using-workflows/events-th...

I remember seeing a couple projects shared before, using this technique to scrape sites with GHA

I came to suggest the same thing.

I use their cronjob functionality to ensure my docker images are built daily and therefore in theory secure.

Just note that they’re not guaranteed to be called precisely on time, e.g. my “every 15m” CRON job will be called every 15m _at best_, in practice… twice per hour.

This works perfectly for my case (content syndication for https://potato.horse), and I’m pretty happy with GH actions for this kind of stuff, but if you need something more precise, you might want to look somewhere else.

At least for a while before I moved to self-hosted systemd timers, if you were running jobs near 0:00 UTC, the delay was often so long that your job wouldn't run at all. I had weeks of jobs literally not running at all before bailing on GitHub actions for this sort of thing. It was disappointing, but I'm now happier with my current setup.

> Just note that they’re not guaranteed to be called precisely on time, e.g. my “every 15m” CRON job will be called every 15m _at best_, in practice… twice per hour

Is the spread really that egregious? That's essentially a 50% failure to trigger at all, like I don't think you can call that 15 minutes so much as 15±10 minutes lmfao

IIRC AWS EventBridge is also not guaranteed to execute on the exact minute, but in my experience running a small job every 5 minutes only had about a 30-40 second delay at worst.

Just checked the logs: it used to be 2 per hour, but now it has improved to 3-5, so closer to the "every 15 m" rule. Again, not a big deal in my case and probably something that's being worked on.

AFAIK, scheduled GitHub workflows stop running after a while. But when that happens, GitHub will send you an email with a big green “Continue running workflow” button.

I use crons to keep my Docker containers fresh, and have never hit this. But the cron commits to the repo, so I wonder if they’re flagging repos with crons but no commits recently?

Definitely... It seems to be in 4-6 week timeframe... I'd thought about making the cron update references in the repo, but hadn't gone through that as of yet.

sounds like time to make a “hit the big green button” github action

I've got a couple projects like this, that mostly just create bundles of other source projects that I'm not involved with. Creating a windows installer, or docker image for projects that don't have that integrated. It's kind of annoying that they will stop in several weeks when there's no project changes.

> It's kind of annoying that they will stop in several weeks when there's no project changes.

Can't you configure a GHA which commits nonsense every month?

Yeah... I will probably do that, with my daily check action (update a last-check) text file or something.

Can you not trigger an action via a local cronjob and the API?

But then I need a second system to do that.

I wonder if I can make a Github repo with an action that commits to another repo on a cron. Or itself?

Actions won’t run on repos with no activity for 60 days [0]. So make sure your action commits something to the repo to keep up activity.

This seems to meet OP’s use case as they have to commit state anyway.

[0] https://docs.github.com/en/actions/managing-workflow-runs/di...

https://simonwillison.net/2020/Oct/9/git-scraping/

Simon Willison has a bunch of examples of scraping sites with GHA and storing the results in a repo. But you can use the same technique without the storing part if need be.

This works great! I do this in one of my projects if you need a reference.

https://github.com/patrickgalbraith/rageagain/blob/master/.g...

AWS Lambda and Azure Functions both support timer triggers and probably round down to $0/ month if your job only runs for a few seconds an hour

I exclusively use Lambda with Cloudwatch Rules for all the crons I write these days. It just works out of the box.

You should check out EventBridge Scheduler. Just released last month and seems like the go-to solution for such things going forward. I especially like the "run once at this time in the future" feature.

Is that the same event source that's been available to the aws::serverless::function transform for awhile now?

That’s neat! I remember hacking that together at an old job using step functions with a “pause until Timestamp” step.

Yeah, this is exactly where my mind went too. I think it’s a great example of not jumping to technical solutions too fast. The poster was so locked in on specifically cron as a solution that they forgot what they really want is code or jobs to run on a schedule and then missed alternate solutions.

Yep and with the CDK it's not only easy to spin up, it's super easy to connect services to it.

I wrote a notifier for when a local business has open appointments. A quick fetch every day and connected it to SNS seamlessly. Now I can easily add family members if they want the notifications and they can easily unsubscribe right from the email. All within the free tier.

You get something like a million lambda executions per month in the free tier.

Raspberry Pi at home if you don't have a home server already. Or some low spec cheap vps.

Yes yes yes. I came to post the exact same thing then saw this =)

Eh.

I tried to do Chromium/Puppeteer based scraping this way.

Building a Dockerfile took ages due to the low compute. (Rust was a non-starter).

I also had (foolishly) only bought the Pi with 2GB instead of 8GB so RAM was an issue.

Disk was super slow.

I'm not sure how viable this is, especially with how hard it is currently to source a Pi, let alone its computation/memory constraints.

That's really deviating from the nature of the "cheapest, easiest way to host a cronjob" question. If the OP has that kind of requirement, he won't get good answers.

You really don't need docker for this.

It's true that you don't, but I can see the advantages.

I have a Raspberry Pi that is natively running a scraper using headless Chromium and cron. It works great, except....

I ended up needing a virtual framebuffer. I got it working on the Raspberry Pi, but I got a new workstation and wanted to edit my script and test it there. I got cryptic errors that I needed to debug to understand they were framebuffer issues, then attempt to recreate the setup that's running on my Pi, then debug that.....

My first mistake was not writing down what I did in my README, but a Docker image would have saved me a ton of time here.

Why couldn't you keep editing your script in your Pi?

Isn't proposing Docker as a solution to this going nuclear?

I think there are some many cheaper things you can do to solve this.

For me this is absolutely no justification for Docker use in this scenario.

The script takes 60+ seconds to run on an RPi and fewer than 10 on an i9, allowing for significantly faster iteration.

Doesn't really answer any of the questions of what docker actually does to help here, other than make everything slower and more complex.

You can use the "browserless" docker service which contains a headless chrome browser in a docker container. It also supports puppeteer and playwright connect api. Works flawless! I use it in combination with n8n. All on a raspberry pi4b (yes, I got one recently)

> You can use the "browserless" docker service which contains a headless chrome browser in a docker container.

A lot of websites can detect the IPs from this and block it, basically almostly like a CAPTCHA.

I had other needs like Postgres, etc.

Scraping data is one thing, actually doing anything with it is another. I quickly hit the limits of a $50 RaspberryPi 4 or whatever they're going for on Amazon these days with the gouging, etc.

> Building a Dockerfile took ages due to the low compute

Why compile anything on the raspberry pi? Cross-compile on a machine with more compute (like your laptop, desktop, phone, or ec2 instance) one time, and then transfer the compiled binaries or built docker image over.

> Pi with 2GB instead of 8GB

For headless chrome, that should be enough unless you're doing other stuff with it. Unless you mean for compiling stuff, which as before can be done elsewhere.

You're mentioning the worst possible scenario.

First of all the answer recommends to use either a Pi or a VPS. If a Pi doesn't cut it for you, just switch to a VPS with sufficient specs for your requirement. Problem solved, now it's viable.

Besides, a significant part of the web can be scraped without resorting to a heavyweight browser such as Chromium. It should always be the last resort. Even if you have to evaluate Javascript (in case of SPAs), there are much cheaper solutions than Puppeteer (JSDOM being an example) which can get the job done most of the time.

As to Docker, I fail to see why you would need Docker for this kind of job, unless you don't know how to do it without Docker.

When there is no portability requirement, the costs of Docker easily surpass its benefits.

...

So no, it's not that what's being recommended in the answer is not viable. You're doing it wrong. Either you're using a Pi when you need to use a VPS or you are introducing unnecessary layers.

i run it on my NAS at home, one of the reasons is that it does not have a PUBLIC IP of one of the providers that is blocked on my scraping target.

A use case I've needed cloud services for is web scraping. Sites which IP ban web scrapers will still allow scraping from major cloud provider IP space.

How are sites detecting and banning once-an-hour users? Yes, if you want to spider an entire website continuously, you could get IP banned, but if you're making 24 (or even 96) requests a day to visit one site and check some data, your traffic is indistinguishable from baseline.

I've seen bigger sites block requests from ip ranges that belong to hosting providers. Don't know about cloud providers, but doing it from home with a residential address has the highest chances of success. You might still end up in the cloudflare verification page though sometimes.

Although they still seem to be out of stock almost everywhere. (Looking for a Zero W)

Take your choice of other random SBCs off Aliexpress for $30-40. They all run Linux well enough for headless, short cronjobs.

My recommendation:

- (free) set up google sheets (not sure if this is required but that's what I do)

- (free) run an scheduled appscript bound to that sheet with regular intervals

- make it scrape the page (it's just javascript, you can do get requests, etc)

- Have it scrape a webarchive of the page you want to compare (not need to persist state or run a database, etc) and have it check the differences with the current page

- Have it send you an email if there's changes

It's totally free, takes no time to set up, and is relatively effortless. It can even send you alert on failures etc. I do something like this for my reading tracker where I crawl various sites (like amazon, etc) and RSS feeds for new releases of books/series I read and collect them in a dynamic spreadsheet.

good memories :-) I started programing with google apps script.

I think it's a great starting point for students because it's totally free, abstracts over the coplications of oauth (for small things I still prefer it because it's so effortless) and you can combine google apps in fun ways. Considering many students live out of gmail + google docs, it's great

And having a GUI version of the app makes interacting with other students easy bc they can always edit the google sheet (aka database), forms, gmail, etc

I'm not sure what you mean when you say there is no easy solution... procure a small vm on Linode or your choice of provider for like $20/mo. SSH into it, set up the crontab to run your web scraping script. I could literally have that set up and running in 15 minutes flat.

EDIT: after reading the other solutions in this thread I'm blown away by how much people are over thinking and over engineering this. KISS people.

Another tip is that Linode isn't the cheapest out there. I'm really happy with the pricing with Hetzner.

However, to your point on KISS, it's not really about that. Leaving a server on 24/7 is pretty cost ineffective if all you want to do is run a short job once a day.

I use AWS Lightsail for a similar task, the node is pretty slow but it's only $5/month.

It's reasonable to call this the "easiest" way. It's certainly not the "cheapest, easiest" way, which is what the poster asked for.

Many of the other options listed in this thread are free or within a rounding error of free. Some of them equally are not the "cheapest, easiest" because they're complex. But some of them pretty clearly meet both criteria (GitHub Actions comes to mind), and some of them might meet the criteria depending on what somebody has familiarity with (if you're already used to deploying code to any FaaS provider with a free tier, that'll likely be the cheapest/easiest way).

The way I look at it: If you have an existing VPS already hosting things (and what geek doesn't?) then adding a cronjob to that host incrementally costs essentially $0 and is pretty much the easiest thing you can do.

Just to say it out loud: there are plenty of geeks w/o a VPS, either because they hack on things that don't generally need a server (like hardware, or radio, or any number of other geek pursuits) or because they've historically used local gear for their compute tasks.

If you have a VPS, I agree, adding a cronjob to it is both cheap and easy. But it's silly to assume that anybody who is a geek has a VPS already.

As one of the sibling comments points out, you could set this up on a GCP free tier VM.

Why is running a single cloud VM in 2022 so terrifying?

Overthinking it?

Your solutions costs $240/year.

There are solutions out there that would probably bring that under $20/year, and be more reliable than your VM that falls over when it's full of logs (at some indeterminate future date).

10x+ savings with greater reliability isn't "overthinking it", it's a greatly improved solution.

Cheapest linode is $5/month. Not sure why you'd need one that is $20/month

And LowEndBox has a reasonable list of $1/month servers. And that's not as low as it goes either.

NAT VPS. Under $10/Year

I charge my clients $550 an hour, so a $240 a year solution which takes 15 mins to set up is what I would choose over the time consuming alternatives offered.

I think you are just more used to creating a VM and running crons from there. The other alternatives aren't necessarily time consuming, you just aren't used to them

> more reliable than your VM that falls over when it's full of logs (at some indeterminate future date)

If this is an issue, I think all other solutions have similar risks: account banned, company acquired/closed down, forced update to new API/system, to a point that I think I'd take the bet on a VM living longer in most cases

I do the same thing (as well as a couple of small docker containers, for example vaultwarden) on a GCP free tier VM.

Works great.

Even better that $20/month on linode is a GCP free tier instance. For this case, I bet it would be free.

https://cloud.google.com/free

Until Google kills it

$20/mo is ... high. I run a commercial website (low volume) with a java app and mysql for about $6/mo on Digital Ocean.

And a few funsies "droplets" for that much that host my resume and other low volume, but open to internet, sites.

They all run cron.

How much could a banana cost? $10?

funniest show ever

as the comic strip says, “Here’s a nickel kid, buy yourself a better computer”

hell, you can also just buy a cheap rPi clone and plug it into your router

I came here for this

cheapest linode is $5. If you want to save more, vultr.com has an ip6 only for $2.50.

These are just the "reputable" places, there's tons of small players where you can get a tiny VM for $10 for a year.

You can get a VM for way less for simple needs.

I have a VM from those lowendbox providers from a few years ago that costs $4 a YEAR - static IP, 256MB RAM/512MB virtual, 10GB storage, U18.04

I do not use it for anything serious, but it actually has had 99.9% uptime.

Again, with cheap providers there is a good chance they go belly up but for cronjob that should suffice.

For more serious cronjobs on the budget Hetzner is nice.

Schedule a cron job: https://developers.cloudflare.com/workers/platform/triggers/...

Send an email (for free): https://blog.cloudflare.com/sending-email-from-workers-with-...

There is no way you can do that reliably without a K8S multi-cloud cluster.

At this point I don't even know if this comment is satire or not.

I don't see how it's satirical. If you need a job to run, you need to handle the case of transient node failures. K8S helps alleviate this issue by allowing the job to be launched on any of a choice of nodes. Sure, cron is fine if you're just looking to rotate logs on the local machine, but if you want to reset your nuclear watchdog timer, you may want something more reliable :)

If you need a job to run, you need to handle the case of governments turning against you. A huge nuclear capable military and a top notch spy agency helps alleviate this issue by allowing missiles to be launched on any belligerent nations.

I best add a nuclear capable military to the 2023 budget. I know we need to look after every penny, however, the increased reliability is surely worth it.

Just use AWS MAD.

I'm sensing a need for a blockchain here.

These replies made my day. Thanks guys.

Not just a multi-cloud, but also managed kafka and a managed db across a multi-region deployment. Regions can fail, your job needs persistence and you need to maintain the state of the message in the db..

oh!! have you also thought about the right queue semantics? Do you want your job to trigger exactly one, at least one or at most once? What happens if the previous job is still running when it is time to trigger it again? Is it 15 minutes from start to start or end of previous to start of next? How about metrics? Is your process idempotent?

So, the right answer is 42 unicorns.

not so fast... young unicorns !! Where are you going to log the nasty errors when it fails.

You probably want to setup a self-hosted cloud-based remotely-managed splunk and ensure someone does the maintenance/upgrade for you (those can be challenging)

Have you properly setup your pager-duty ? I wouldn't want to miss one of the cron execution !!

It'd be literally impossible to deploy any of this lightweight, cloud-native architecture without at least 14 Terraform modules as well, maybe more (just to be safe).

I cannot unthink of

It's the future (2015) https://circleci.com/blog/its-the-future/

GCP Cloud Run with a schedule set to call the HTTP endpoint once an hour.

Cloud Run will run any docker image, and give it a HTTPS URL. Scales to zero, allows running in the background for up to an hour.

The benefit over FAAS is that you can run anything you want in the container, including multiple processes.

Also much prefer cloud schedule over AWS version.

I've just created a similar setup last week: 1. Cloud function, uses Puppeteer 2. Cloud scheduler for triggering cloud function 3. Cloud storage for data persistence

Costs around 1 Euro a month, I had a similar setup for free in Heroku before.

I love Cloud Run, but can they please tie these two things together for me? It's a radio button that says "On schedule" and an input for the cron syntax.

If you fit within the language/runtime restrictions, Cloud Functions and App Engine also both can be invoked with the same scheduler setup. We use Cloud Functions for monitoring all the time (schedule to call a function once per minute or so that scrapes some metrics from a third party API and pushes to a Prometheus gateway). Costs us something like $0.12/month and has been extremely reliable. We also run apps on App Engine when we can and there all you need to do is add a `cron.yaml` and you can invoke endpoints there easily and practically for free.

Hey, this is still a WIP, but I'm building [redacted]. We wrap an open-source workflow orchestrator Temporal that offers "durable execution" with very detailed execution tracing and a bunch of recovery mechanisms. We offer a fully-managed Temporal cluster, hosted runtime (so you don't need to worry about provisioning/scaling workers), and offer version control integrations, CI/CD and dynamic config out of the box - so you can go from code to running workflows with no infrastructure overhead. Think "lambda for workflows". Feel free to get on our waitlist!

Here's how a cron job implementation would look with Temporal -

` func Subscription(ctx [redacted].Context, userUUID string) error { free_trial_on := true

  for {
    if free_trial_on {
      // Sleep until trial period passes
      [redacted].Sleep(ctx, days(15))
      free_trial_on = False
    } else {
      // Charge subscription fee
      [redacted].ExecuteActivity(ctx, ChargeSubscriptionAndEmailReceipt, userUUID, frequency).Get(ctx, nil)
      // Sleep until next payment is due
      [redacted].Sleep(ctx, days(30))
    }
  }

} `

Just wanted to say that I’m a big fan of temporal. Looking forward to taking a look at this product and seeing how it compares to temporal cloud offering

Just to know... why is it called [name redacted] instead of orchestration.cloud ?

They are two very different concepts which I see very often used interchangeably. Temporal is clearly about orchestration.

as a former temporal employee, its cool to see this, but how did you pitch YC on being essentially a Temporal wrapper? whats the moat?

I have been using https://visualping.io/ for precisely the problem you are describing.

Alternatively, for a more generic solution to “easy free cronjobs”, you can write some php page on a free php hosting service and point visualping to that page. Cost: 0/year

Lots possible solutions. One of them is using Cloudflare Workers, it supports the scheduling parts and storing the previous state. Taking into account the check frequency you describe, I guess it falls within the limits of their free offering.

Just use a 2.5$/month instance on Vultr for your cronjob, and plain old sendmail for your emails (they will go in your spam folder but at least you don't have to waste time setting up emails).

I recently utilized Netlify for this. My build was actually my scraper and I used their Scheduled Functions (cron) to trigger this build every so often.

https://www.netlify.com/blog/how-to-schedule-deploys-with-ne...

Came here to say this. Adding on to that, Netlify Dev has been quite useful to run and debug these functions.

https://docs.netlify.com/cli/get-started/#get-started-with-n...

I've had good luck implementing this "git scraping" pattern: https://simonwillison.net/2020/Oct/9/git-scraping/

Might be more precise to call it "GitHub scraping" as you set up your script to run on a schedule on GitHub Actions CI and keep its state by committing into the git repo.

We have our scripts running on GH Actions end by hitting the Slack API to notify us with a message.

https://github.com/dgtlmoon/changedetection.io

There's a hosted solution, or you can self-host it

This ^

Does exactly what it says and gets bonus points due to self-hosted or as-a-service.

yes!!! changedetection.io is awesome, thanks for the nice words (project owner here :) )

Easiest: Linux VM with cron. Pay for it, SSH into it, crontab -e, add your line, save file. Takes 5 minutes to set up. Moderately reliable. $1-$5/month depending on the provider

Cheapest: Any CI/CD service provider. Takes 20 minutes to set up. Not very reliable. Free

Next cheapest, least easy: complex proprietary serverless cloud functions to automatically trigger a task on a schedule. Takes an hour to set up. Most reliable. $0.30-$1.5/month

  # Hosted Cron Jobs
  -------------------------------------------------------------
  | Solution          | Difficulty | Reliability | Cost       |
  -------------------------------------------------------------
  | Linux VM w/Cron   | Easiest    | Moderate    | $1-$5      |
  | CI/CD Provider    | Moderate   | Low         | $0         |
  | Serverless Cloud  | Hardest    | High        | $0.30-$1.5 |
  -------------------------------------------------------------

From Dictionary:

Cheap: low in price, especially in relation to similar items or services.

Easy: achieved without great effort; presenting few difficulties.

Because "easiest" AND "cheapest" are logically conflicting, my answer only applies to "easy" ways of hosting cron jobs. My answer also holds true for background tasks that are not always scheduled.

The two tools I rely on are:

1. Pipedream

2. Windmill (open-source pipedream)

Both of them let me define functions that need to be run on schedule. Also functions that can react to real world events (webhooks). I can code in Go/ Js/ Ts or Python. Everything is version controlled, stored in Git.

For Pipedream, you pay a premium for managed service. Windmill can be hosted on a $5 machine.

I'll check out your suggestions. I'm not sure you explained why cheap and easy are logically conflicting, though.

https://cron-job.org

I came here to add this very same service. Very nice UI. Simple. Solid so far.

For hobbyists/non-critical work, this is the answer. I've been using it for a while now and never had an issue. But as with all free services, it might be here today then gone tomorrow.

cheapest would have to be running it on the computer you made this post from .. assuming its always on and the delta to your power bill is negligible. as others have suggested if you have an RPi or some other low power SBC laying around, that would be ideal.

i'm sure the major cloud providers have cheap / free tiers for this kind of work, but quite frankly i've been burned by run away pricing tiers too many times to ever consider using cloud again for a personal project .. so unless this is getting funded by a client try to host your own.

Sounds like a job for serverless functions, e.g. in Azure [1], though I'm sure every hyperscaler has that. The Azure version even uses Cron syntax for the scheduling.

[1] https://learn.microsoft.com/en-us/azure/azure-functions/func...

I came to post about this, its super simple too

Try https://render.com

The UI is excellent for this. You can probably find cheaper on Fly but probably not easier.

Even easier, maybe more expensive than Render:

https://www.zeplo.io/docs/schedule

You just hit a URL like so and it's done. zeplo.to/your_url.com?_cron=5|14|||*

I co-founded Airplane.dev, which a lot of companies use as a much simpler, serverless, user-friendly alternative to cron: https://www.airplane.dev/schedules

We have companies running hundreds of concurrent schedules on our platform. Send me a note at ravi@airplane.dev if you'd like to chat about it.

All free:

* free-tier vps on GCP or Oracle Cloud

* lambda job on AWS

I have a cheap VPS I use for other things and just run my cron jobs there.

I don't know about the others, but AWS is not free and it's not cheap. There are ways to set alerts on budget overruns, but there is no foolproof way to limit monthly spending. Eventually you will configure something that goes out of control, or I was able to conceive of a few scenarios where a malicious third party could rack up charges for, e.g. S3 egress costs. Good luck contesting the charges more than once. I shut everything down and pulled out of AWS. I believe that Azure's free trial featured a hard cap, but still requires a credit card to open your account.

The lambda free tier on AWS is forever, and if you’re kicking it via cloud watch vs an api gateway it shouldn’t be possible for it to overrun.

Lambda on AWS is basically free for personal crons. I ran lots on there until I moved everything to a personal k8s cluster for some more granular control over scheduling.

The free tier on Oracle Cloud is nuts. 200GB block storage, 10gb object storage, 24GB RAM 4 core ARM VM, 2 x86 VM's 1GB RAM each. From my experience, it's also really hard to get charged since you have to upgrade your account from a free tier. I haven't been charged one cent in several years.

Yes, it's the best one for general compute I've found. https://paul.totterman.name/posts/free-clouds/

Retool Workflows is the way to go https://retool.com/products/workflows (see the 4 min demo on their site)

Flexibility to write custom code and alerting logic, but no headache of managing your own infrastructure

Disclaimer: I previously worked at Retool :)

Retool Workflows PM here. Thanks for the mention. We just launched in beta and would love any feedback. You can authenticate to your DBs/APIs, write any logic in code and then schedule it or trigger from a webhook. We also price on data throughput (not tasks/steps/etc) so you can use it economically for large jobs. Ping me a note on jamie@retool.com if you give it a try https://retool.com/products/workflows

I recently had some smallen private projects and I reasearached and tried out a lot of services. I needed cron jobs to run Node.js scripts properly.

My use cases have been: - Web Scraping of some websites to check stock availability of a product (4 times a day). - Web Scraping of a website to get an apppointment at a specific citizen office (every 5 seconds). If success then send a Telegram Chat Bot message to me.

The second use case was a bit tricky, because my IP was kicked frequently from the server, I tried out what the sweet spot timing for the ping was and my setup (all jobs with different IPs) was the following:

1. setup on my Raspberry Pi in combination of the package PM2 (PM2 is a daemon process manager that will help you manage and keep your application online 24/7)

2. setup on Render (https://render.com/) Advantage here was the very convinient, quick and easy setup. I was able to link my Github Repository to it and the Cron Job would re-build the program automatically whenever a new master version on GitHub was available. I used an in-memory TypeScript job scheduler library in my project that repeatedly executes given tasks within specified intervals of time (e. g. "each 5 seconds") until 7 PM and I let the scron job service execute the script again at 7 AM in the morning. Debug Print Console inclusive.

3. setup on EvenNode (https://www.evennode.com/docs/cron-jobs) Here the upload of my node.js script was done via FileZilla to the FTP-Server of EvenNode. Linkage to GitHub Repository is also possible but since I am not that familiar with the private key setup at GitHub I chose the FTP-Server option. I used an in-memory TypeScript job scheduler library in my project that repeatedly executes given tasks within specified intervals of time (e. g. "each 5 seconds") until 7 PM and I let the scron job service execute the script again at 7 AM in the morning. Debug Print Console inclusive.

Both online Service were very cheap, paid cents only. The Raspberry Pi solution was for free (only energy consumption). I can highly recommend render.com and evennode.com for cron jobs.

I use uptime robot connected to a serverless function, with the great property it can notify me of outages on top of being a scheduled http call. I think the first 50 are free!

https://uptimerobot.com/

GitHub actions are another option

These days you can do it entirely for free with GitHub Actions, here is one simple example if you want to e.g. schedule a simple Python script running as a cron job https://medium.com/the-prefect-blog/scheduled-data-pipelines...

Can someone please enlighten me on what is the purpose of using an external Cron runner here? If I need to run the same job(s) for a long time, wouldn't it be simpler and cheaper to run a basic $5 VM on something like Linode?

The motivation is some of the "serverless" offerings may be near free if you are only firing this say once a day and the script is very fast to run.

Yes maybe, however the guy said he wants to run once per hour. On top of that he will certainly need some form of persistence in order to do diff. Not to mention that some of the serverless solution suggested in other comments have short execution time limit, which if he happen to scan a large site, that would be insufficient. Yes maybe not everyone is willing to pay $5 per month, but in this case I think it strike a reasonable point between cost, speed, reliability and ease of use.

rely dislike those 5$ VM comments

You can checkout https://cronhub.io (I run it). Behind the scenes, it wraps a lambda function with EventBridge (so you can set schedule) and adds other goodies like notifications/alert (slack/email etc), metrics, notifications if job runs slow etc.

Not free but basically you wrap your job in http endpoint and then we take care of the rest.

I think OP doesn't want to keep a machine running permanently. If you do that, adding your own crontab entry would be trivial as well.

Possibly. We do have paying customers who are using it so that they have better tracking, notifications and guarantee that the job is running (with metrics etc). But I hear you. For some devs, it may not be critical. I do have plans to allow adding your own code directly so that you don't have to come with an http endpoint.

I suppose with this solution you could also run one of those free tier machines that spins down when not in use, or a lambda function, etc

Cheap vps running urlwatch sending alerts with pushover. Turn on unattended upgrades for the server. The one time pushover fee is well worth it.

+1 for urlwatch. I run it on my pc with Docker (restart=unless-stopped), and Discord integration on a personal Discord channel with alerts enabled. Takes two minutes to set up and is online whenever I'm online.

I've never known about urlwatch, I always rolled my own in python (and then copy/pasted for the next task)

there it is an apt-get away

Thanks!

https://visualping.io

Mergent (YC S21) https://mergent.co -- our crons & background tasks are free for the first 1k invocations per month and pay-as-you-go after that.

There are two ways to set up a cron job: the dashboard or via the API if you need the flexibility, e.g., dynamically creating/starting/stopping crons.

We use HTTP webhooks to make the outgoing cron requests, which makes it easy to integrate with a wide range of services and platforms, including Vercel, Lambda, etc. We're also working on adding local execution so you can have even more control over how your tasks are run without using webhooks, but that's still being built & tested.

Feel free to reply here or email me directly if you have any questions: james@mergent.co

It amazes me that for such a relatively simple thing everyone's knee jerk response is THE CLOUD!

Yeah why can't my phone or tablet do something simple like this.

Your phone or tablet is not always on

It’s the easiest and cheapest way unless you have an existing raspberry pi or something. Your personal devices are probably not always on.

Google Apps Script

https://developers.google.com/apps-script

It's free, so pretty cheap.

You can set up a schedule to run the scripts. Has easy access to Google APIs (Gmail).

Very powerful and simple solution I've used for years.

EventBridge + AWS Lambda

It's cheap as in free, thanks to the generous free tier.

isn’t the free tier only for one year?

No, you get some free usage in perpetuity

Or until they change the terms & conditions...

Some services have a small "always free" tier too; S3, DynamoDB and Lambda all have enough free usage available for personal projects.

I find the meta implications on this thread quite interesting:

How cheap can you run this cron job, the basics:

- 8,765 invocations a year

- Lambda: $0.20 per 1M requests

- Cloudwatch Events: $1.00 per 1M requests

$1.2 / 1,000,000 * 8,765 = About 1 cent

Is anybody going to host that as a service for this price? Absolutely not.

Easiest? Probably https://repeat.dev

I set up a Raspberry Pi on my local network because every year I end up with dozens of little use cases for a Web addressable, always-on machine where I DO NOT need redundancy, uptime guarantees, backups, CDNs, high bandwidth needs, etc.

I want to offload this, but any time I search for an option, they're overcomplicated or overpriced. Maybe there just isn't a market for it?

...maybe I don't want to offload this. The Pi and SD card was like $35, the power is basically zero, and the Internet is basically zero.

Check out https://apify.com (disclaimer: I work there :)) - a platform focused on web automation and web scraping. You can easily build any (but mainly NodeJS or Python) project there as a Docker container and schedule it in a minute. See my 2 min. video - https://share.cleanshot.com/osSygJ

Priced: $0.25/GB-hour + data transfer and storage

Probably not the cheapest, but I created beew.io to be the easyest.

https://beew.io/

You can create cron jobs over HTTP, both recurring and one-time:

Node SDK: https://www.npmjs.com/package/beew API Reference: https://beew.io/api

You can also create directly trough the UI.

Let me know if you need any help

Google Apps Script and Browserflow are two of my favorites. Getting a GCP or AWS VM is a solid approach too; both have reasonable free-tiers. And as an alternative to cron, the Python `schedule` library is nice too. Finally if you don't mind some down time, running it on your laptop might be fine. Supervisord is a solid tool for keeping jobs running / restarting them on error.

I'm actively using all of the above approaches.

How about Kubernetes cronjobs on Spot VM on GKE autopilot? https://cloud.google.com/kubernetes-engine/pricing If your job only consume 1 vCPU/1GB memory, it costs about $10/month. If those jobs only run 1/100 of a month cost should be $0.1/month.

okay, why can't you just plug it in /etc/crontab ? If you're just scraping, you don't even need a public ip, and since you're writing this, I assume you own a PC..

Because not everyone has a pc/home server they can just leave on 24/7

That seems unlikely to be honest.. just grab an old one if you don't want to use your main one for it.. but if it's cron, its probably not so important that it can't survive missing a single run due to some reboot or whatever?

Electricity prices at the moment mean running an old PC 24/7 isn't the no brainer it used to be.

Not a no-brainer for sure, but depending on what PC it is, the costs can still be really low. Assuming 50 W power draw (you can get this much lower with cheap or free e-waste PC's like old Atom NUCs or thin clients, or a low-power laptop like an old MacBook Air) you're using 1.2 kWh per day. Even at current extreme electricity prices this will not really break the bank for most people.

Better, forget about running 24/7 and use rtcwake(8) to wake it up for an hour at night to perform the job and then go back to sleep. This will bring costs far into the negligible.

This is exactly why I am building https://repeat.dev, create a scheduled task (cron) and do whatever you want quickly and easily. For a cron, we got a https://repeat.new/cron template.

I use Render.com for this - including some scripts like "Email me to invoice a client on the 1st of every month"

I'm still using the old way: "crontab -e" inside the same EC2 where I run the app. In case of failure I automatically send an email to me. This is the easiest scenario in case you already have an EC2 for some other things that is already running. Instead, from scratch (no EC2), I'd probably use Lambda.

Out of curiosity and for the sake of completeness, by failure, you're probably talking about an error that the script itself can detect? What about a scenario where the EC2 goes down (silently)?

Yes I mean failure of the cron script itself. If the EC2 goes down entirely there are other alerts you can set up on AWS, such as Cloudwatch or similar.

Can't you host this on hardware you are running in your house?

Agreed that most of the solutions are still tough to implement in production. Plus, a lot don't have built-in logging, error-handling, notifications, version control, package management, etc. Those are the "gotchas" that you won't notice until things inevitably go wrong.

It's worth checking out Shipyard (https://www.shipyardapp.com). I'm the co-founder and designed it to be the easiest way to deploy scripts/workflows in the cloud. You can schedule any code to run our platform (native support for Python, Node.js, Bash) and build reusable templates.

Our free dev plan allows for 10 hours of free runtime per month which is plenty for most use cases. If you want more or need webhooks/API, that starts at $50/month.

Feel free to contact if you want to learn more. Email is in my profile.

Cheap? Elephant in the room but going to mention it anyway: shared Apache host usually has cron+php+db.

https://visualping.io/

Your own browser.

https://chrome.google.com/webstore/detail/distill-web-monito...

If that's all you need checklyhq.com (not affiliated with them) free plan should do it!

Netlify Scheduled Functions for basic TS Code.

It's free.

https://docs.netlify.com/functions/scheduled-functions/

https://www.abstracloud.com/jobs

It is definitely the easiest way for Python code. Just paste your code and thats it

Full disclosure: I made this product. We are a YC company

Basic plan is $35/m which is much higher than alternatives discussed in this thread, isn't it?

Especially seeing as you can outright have a full, tiny VPS 24/7 for $2.50/mo or $30/yr from Vultr. I suppose there's a value add in terms of not necessarily needing to setup your own install, but if that's something you're comfortable with I don't quite see the point.

You're right, this is more expensive than VPSs alternatives.

We assumed commercial use cases for that price so the benefits of not having to deal with servers complexity + access control + etc.. are quickly paid back.

For hobby projects, we are thinking of creating a free plan

If you have a GCP, AWS, or Azure account already and know Javascript, you should be able to whip up a serverless cron task real quick. IMO the simplest setup is probably GCP via Firebase (which makes deployment super simple once you install the CLI): https://firebase.google.com/docs/functions/schedule-function...

You won't spend much at all, it's fractions of a penny per call and I think there's a free tier.

My scenario was triggering a Firebase backup once a day, but being to cheap to upgrade to the paid plan, that does it. (There also isn't an easy way to cap costs with Firebase). I actual backup is handled by a serverless function (in my case hosted by Vercel). There are other services that have somewhat generous free tiers, but be aware that those are usually quite restrictive on run time. For triggering the actual cron job I first tried Github Actions, but found it to be rather unreliable and ultimately settled for qstash by Upstash.

I am in a very similar situation, what I ended up doing is use https://uptimerobot.com/ to monitor a specific route "HEAD https://example.com/update", which will update the data. It sends a HEAD request to the given path.

0 code on the cron side (cannot be simpler), and if you only monitor a single route per hour is free as well. AND you get a free UI on how the cron job went :)

Azure functions, as far as I know, have milion executions for free. You can use their "timer teigger" which is a fancy way of saying cron job.

More info:

https://azure.microsoft.com/en-us/pricing/details/functions/

https://www.serverless360.com/blog/azure-functions-triggers-...

Temporal and its ”schedule” feature is great - it handles backfill and you can set various policies [0].

We self-host, but they have a GA cloud offering now. [1]

As long as you use js/ts, go, python, php or java that is. For some jobs we wrap process execution - e.g running a powershell command.

[0] https://docs.temporal.io/concepts/what-is-a-schedule

[1] https://temporal.io/cloud

I've had good experiences with Google Cloud Scheduler. https://cloud.google.com/scheduler

You can run jobs with the granularity of a minute for free with Cloudflare Workers[1].

[1]: https://workers.cloudflare.com

https://www.webcron.org/en/prices

One of the cheaper options and probably one of the older options.

There are plenty of budget VPS hosts out there where you can get something for <$20/year. They're not good but they are good enough. Check out lowendbox.com.

Hetzner are $45/year, or even less if you don't care about ipv4, and they are great.

It might be more than what you need, but your needs will increase. The above cost will not.

If this question came up at work, I would recommend a Lambda function that sends email via SES and is triggered periodically by a CloudWatch event. It would be cheap and require basically no admin overhead once set up.

That said, there are probably different forces at play if this is a personal infrastructure question: while cheap is good, if you aren't fluent in AWS it may be a slog to set up. If you're not, easiest thing is probably a real cron job run on a cheap VPS.

Check out scheduled jobs feature of Superblocks (https://www.superblocks.com/product/scheduled-jobs). We've got a robust free tier (https://www.superblocks.com/pricing) and support Python, NodeJS, REST, GraphQL and a heck ton of DB integrations.

How many do you have? There's fixed costs and variable costs, after all.

NixOS-configured systemd timers work great. Code them in bash, Go, Haskell, whatever. If you like polyglot (mostly to leverage any community's open source apps), NixOS is the best.

It isn't the cheapest or easiest for a single job. It's not hard to deploy to a random $5 VPS though (I use a Digital Ocean droplet). But once you have a box to deploy to, the incremental cost is as cheap as it gets.

You can checkout Deta (I work for Deta).

Free, the job timeout is 10s but can be extended if needed, and minimum resolution is run every minute.

https://docs.deta.sh/docs/tutorials/cron-guide https://docs.deta.sh/docs/micros/cron

Depends on the job, how long it runs, but could be hundredths of a penny a day on AWS lambda (cloudwatch scheduler accepts cron specs for specifying when it runs)

> It is 2022, so I had thought it would be tremendously easy and cheap, but it seems no solution is easily implemented.

You assume someone purchased hardware (preferably available anywhere on the planet), powered it up, set it up, connected it to the internet, then built the software to handle your very specific task, then put it online to do specifically what you want and for a close-to-free price? And you're shocked this doesn't exist?

It's not as if we've just had a decade of such unthinkable business strategies. Google is still, for the vast majority of people, free (as in $0 per month) service which gives them free email, document editing, and file storage.

(Not Op) Yes. The task is specific but not all that unusual. In a world where startups seem to exist to solve every conceivable problem (including ridiculous “problems” that nobody has), it’s always interesting to find something that isn’t nearly solved.

Personally, I use render.com to host a bunch. They cost like $1 each a month. Super easy to deploy to because it's just a docker container or script.

Free tier on Google Cloud. You can use the free VM instance they provide (1 micro per month) and do an actual cron or use App Engine's cron thing.

Digital ocean functions just release a “scheduled” feature. It’s super easy and has a generous free tier.

I’m using it to issue a daily db back up command for my app.

Any of the serverless cloud functions work well and are exceptionally cheap. Azure has a template specifically for this.

https://learn.microsoft.com/en-us/azure/azure-functions/func...

A pretty easy and at the time of writing free option is to use Dark: https://docs.darklang.com/tutorials/create-daily-job-cron-ha...

Zero infrastructure to manage, only write your script (in Dark, for that matter).

How short?

How much local compute do you have?

I think people underestimate dynamic DNS or a static IP if you can get one for your home for cheap.

I would just defer to Vercel's recommendations: https://vercel.com/guides/how-to-setup-cron-jobs-on-vercel.

Tl;dr: Github Actions for low frequency, upstash for high.

GitLab.com scheduled pipelines. You get 400 free minutes per month and can do whatever you want.

https://docs.gitlab.com/ee/ci/pipelines/schedules.html

I run my cronjobs on OpenWRT router. It's turned on all the time anyway, so it's for free :)

I have a PowerShell script that collects some RSS and posts it to my journal.

Easiest way I found to do that was Azure Functions. Costs me about 35p per month. Mostly for storage for logs as far as I can tell. I'd sort that, but it's literally not worth the time it would take

AWS Lambdas could be used for this, you can trigger them on a time schedule using an event bridge.

what happens if the processing takes longer than 30s?

The limit is now 30x that :)

Lambdas can run for 15 minutes

Citation here https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-...

Using their Python library called chalice was really easy to make cron style lambdas in Python in my experience. https://github.com/aws/chalice

A free SDF account?

https://sdf.org/

IFTT plus URLLooker would be like $12 a month. I would be tempted to use them instead if I made an average coder's salary, especially if it was for a client who didn't have anyone else who knew cron and scripting.

A VPS at hosthatch is still 4.99 I believe.

Sounds like a task for a cheap VPS

Sometimes you get cron jobs with a cheap webhost, so you would not even need to maintain a cheap VPS

Maybe they're out there, but I cannot recall the last time I saw a webhost for less than Vultr's starting $2.50/mo

You can schedule Buddy.works to run according to crontab:

https://buddy.works/blog/introducing-cron-jobs

What ist you question about?

Is it about how to run cron? This is a thing of the past. As you are asking for 2022, look at systemd timers.

Is it about cheap hosting? The are lots of answers here already.

Or is it about how to monitor a site for changes?

Github Actions. I run https://twitter.com/someonegoogled every hour using it. never fails. never late.

Since we are at the topic, what the best way to schedule production grade tasks on a Windows server? I joined a new company and they are using Task Scheduler and it is just awful!

step 1. install Linux

I wish! As a newcomer I have to wait before I start making such (sensible) suggestions!

As someone who always missed failing cronjobs, I am using Airflow with Slack integration. I.e., every time the job finishes, I get a message. If the job fails, I also get a message.

It’s glorious. Post to a muted channel on success, post to a unmuted on on failure.

hello,

you want to watch a web-page - apparently they have no RSS-feed in 2022... that got out of fashion a long time ago:

easy: idk. in a short shell-script do something like:

curl -o /tmp/somefile-<current-timestamp> <your-URL>

diff /tmp/somefile-<current-timestamp> /tmp/otherfile-<older-timestamp>

or calculate a hashvalue of the page & compare ...

or go for the last-modified, e-tag or content-length in the http-header

curl -I <your-URL>

put this file into your systems crontab.

just my 0.02€

If I were going to design a system to do this, I would consider using AWS EventBridge to drive a lambda directly or enqueue a job to an SQS queue that a lambda is subscribed to.

I would suggest anything serverless in most cases. Cheapest is not a great measure. Azure and GCP have good options. Cloudflare too if you just want to do something simple.

I've used GCP Cloud Scheduler for simple cron jobs. It's painless and cheap. You will probably be able to run it in the free tier entirely for low compute jobs.

> Myself I have had to make a short running web-scraping job that, given a change in the site, sends a notifying email. This running once an hour.

locally hosted linux machine (or vm)

I set (and forgot) a Twitter bot a while ago that generates a randomized image and posts it,via GCP with a cron trigger twice a day. It hasn't cost me anything.

I have several cron jobs running on https://www.deta.sh/ . Its free and works well.

A virtual private server can cost only $2/month. If you're looking for task specific, such a simple website monitor, there are services available.

apache airflow and a $5 vps might be the cleanest solution

easycron.com

$24/year and you can basically do anything you want.

All you gotta do is select the desired interval, you tell easycron which endpoint you wanna hit, and that's it.

Or https://beew.io

Any of the managed serverless options with a free tier cloudflare has a free tier but it looks very limited compared to AWS/GCP/Azure

raspberry pi on your own network that updates an RSS feed that is hosted on an S3 bucket. get the RSS updates via reader, outlook or slack.

This is part of the motivation behind https://val.town (I work there!)

There's Apify for this exact use-case, their free credits combined with the pay-as-you-go pricing should be a suitable solution.

Amazon EventBridge schedule -> Lambda function -> SES. This should cost you pennies, assuming you ever break out of free tier.

You get 3 free scheduled functions using firebase, and you can run Node.

To store data you’d use FireStore and/or Storage.

Both free for your use case (probably)

Try Pythonanywhere. Currently, I'm using this to run a trading bot.

Cost min $5 per month and then additional $1 for each additional task

IFTTT is free if you don't need perfect timing. Create the job, a http endpoint IFTTT can reach and you're done.

I just set up something similar myself. Render.com was incredibly easy and intuitive. I think it costs me $1/mo.

I like https://northflank.com/

Not sure about easiest exactly but GitHub Actions has a ton of functionality you can build around a cron schedule

You could configure crond to invoke it for you. Debian already has a cron.hourly facility for that rhythm.

Join something like sdf.org and pay the $60 or so dollars for lifetime membership and you’re all set.

It is 2022, you don't necessarily need a cronjob, you need to know when the site state changes.

https://m3o.com/cron

FasS + writing the code with ChatGPT

Any serverless FaaS offering should have a Timer trigger, which is essentially a cronjob.

Any cheap vps linux box will have cron and mail sending utils installed and ready to go.

Some boxes come with systemd, which has its own scheduler, but not cron. This is not a big deal but I was somewhat confused the first time I saw it.

I'll admit, most of my linux admin experience predates systemd.

Render charges $1/mo for this. I don't know if it fits your use case, though.

I’d go the the simplest approach - cheapest Linux VM you can get and use cron.

cron-jobs.org is pretty good.

I built a cron job utility service with Pipedream workflows because I needed some additional features like sending email report and hooking up to a cronjob monitor like cronitor or healthchecks.

If your job is small enough and not ran too often, AWS is literally free.

Mergent.io has this feature. Don't know if it's the cheapest.

Thanks for the shoutout! For anyone that sees this, the URL is https://mergent.co (.co, not .io).

- James

GitHub Actions does cronjobs for free if the repository is public.

Use a systemd timer instead. On a SBC at home or a free VPS.

I have done this on Hetzner. Cost about 10cent per scan

Lambda function, scheduled by eventbridge

the simplest and cheapest

Maybe something like uptimerobot with a webhook.

Run it as a GitHub actions with Linux runner.

Google App Engine, Google Cloud Tasks, free

Aren’t firebase functions practically free?

Are you just looking for a cloud function?

Even in 2022, I’d still use vanilla cron.

CDK / Lambda / EventBridge

A VM in the Oracle Cloud Free tier.

I run a few cron jobs in my router.

What do you guys use that for?

AWS

Lambda

Event Bridge rule

Google Cloud free tier VM?

can you elaborate on the scope ? how many sites and unique pages ?

Google Cloud Scheduler

Gitlab CI

exactly; 100% free on gitlab-ci; https://docs.gitlab.com/ee/ci/pipelines/schedules.html; been running some for years, no issues

Aws Lambda

qstash by upstash

www.deta.sh

Raspberry pi zero or equivalent. Less than $10 and power is virtually free no matter where it runs.

You might just be a noob.

Reply on news.ycombinator.com