Ask HN: How to monitor periodic short-lived processes? | Hacker News Reader

Ask HN: How to monitor periodic short-lived processes?

4 pointssharmi12 comments

Hi HN,

    We are two founders starting up a business. We need to run multiple programs everyday (around 25) that will be run periodically. They are not always alive though and will often be done in 10 minutes to  an hour.

Unfortunately as a two person company, keeping track of all the programs' statuses is hard.

Is there a way to keep track (and if possible, also spawn them on schedule) of the processes and give regular updates? Like a dashboard of sorts, that can show which programs have been running successfully and which have failed to finish or didn't start?

View on HN

For a *nix box, the simple/historic way has been to run the tasks from cron and append log output from them to text files or the system logs. If you really mean statuses, as in exit statuses, wrap the programs in shell scripts which print out the programs' exit codes.

It's unfashionable these days I think, but still works for me.

You might have to come up with something custom.

Here's one idea:

1. Wrap your processes in systemd services or something similar, where it's possible to determine the status by pinging it.

2. Write a script to periodically (via cron maybe) ping all relevant services, and store the statuses somewhere (file, DB..)

3. Another script (or the previous one) to read the data from 2 and spit it to a locally served webpage.

> if possible, also spawn them on schedule

Cronjobs.

Make use of dynamic tracing and write/customize relevant to needs.

Potential examples:

https://twitter.com/freebsdfrau/status/1238541870612926465 modify to just focus on cron related stuff.

https://www.brendangregg.com/blog/2016-02-01/linux-wakeup-of...

Programming approaches:

ebpf : https://ebpf.io/projects/

Dtrace : https://www.brendangregg.com/dtrace.html

Ylang : https://blog.openresty.com/en/ylang-intro-part1/

Program example: Use execsnoop (https://www.brendangregg.com/dtrace.html ) & add wrapper/filter to feed relevant "live" info to dashboard.

turnkey program(s) per google search:

Cronitor : https://cronitor.io/docs/using-cronitor-cli

Nagios : https://exchange.nagios.org/directory/Tutorials/Other-Tutori...

https://jaytuckey.name/2020/10/30/new-project-coconut-deadma...

https://www.saashub.com/compare-cronhub-vs-zabbix?ref=sideba...

https://www.rundeck.com/open-source

http://www.panozzaj.com/blog/2014/05/02/replace-local-cron-w...

https://github.com/jhuckaby/Cronicle

https://healthchecks.io/docs/self_hosted/

Cronicle and Healthchecks.io seem great. There are exactly what I was looking for. Thank you very much.

Cron is the term you are looking for and there are many options. As another commenter said, there are Linux system level solutions. There are also Cron concepts in kubernetes, aws lambda, zapier. If they don't have a dashboard, I'm sure there is another SaaS that does and plenty of open source solutions for that as well

I do use system level cron. The problem was in monitoring them. Maybe like the other commentor said, one could parse syslog to monitor.

Also curious what the best way to visualize cron status is, for example using timeseries data from kube or a syslog.

Cronitor.io timeline view

ah, we don't care when it runs as much as when it doesn't run, so we slack on failure

cron heat maps

Reply on news.ycombinator.com