Monitoring tiny web services
jvns.ca
jvns.ca
Another neat monitoring thing I rely on is https://healthchecks.io. Anything that needs to run periodically checks in with the API at the start and the end of execution so you can be sure they are running as they should, on time, and without errors. Its free tier allows 20 checks.
It works really well for cron jobs - while it works with a single call, you can also call a /start and finished endpoint and get extra insights such as runtime for your jobs.
It would be nice if it had slightly more complex alerting rules available - for example, a "this service should run successfully at least once every X hours, but is fine to fail multiple times otherwise" type alert.
We wanted to use it for monitoring some periodic downloads (like downloading partners' reports), and the expectation is the call will often time out or fail or have no data to download, which is technically a "failure", but only if it goes on for more than a day. Since healtchecks.io doesn't really support this, we ended up writing our own "stale data" monitoring logic and alerting inside the downloader, and just use healtchecks.io to monitor the script not crashing.
Not sure if you tried it too but https://cronitor.io/ supports more complex alerting rules like the one you describe.
As a bonus, you can also create uptime checks and status pages under the same roof.
Full-disclosure: I work at Cronitor, happy to help if you have any questions :)
This should work if you set the period to "X hours", and send success signals only, no failure signals. In that case, as long as the gap between the success signals is below X hours, all is well. When there's been no success signal for more than X hours, Healthchecks sends out alerts.
I'm guessing you probably also want to log failures using the /fail endpoint. And, the problem is, when Healthchecks receives a failure event, it sends out alerts immediately.
One potential feature I'm considering is a new "/log" endpoint. When a client pings this endpoint, Healthchecks would treat it as neither a success nor a failure, and just log the received data. You could then use this endpoint in place of /fail. Just logging the failure would not cause any immediate alerts. But the information would be there for inspection, when X hours passes with no success signals and you eventually do get alerted. How does that sound?
I saw you make that suggestion on this issue - https://github.com/healthchecks/healthchecks/issues/525#issu...
----
Thinking about it, this does solve the issue as I described it. I do like being able to distinguish the states:
- started, but never finished (no error reported)
- started, and finished with error reported ("crash") (need immediate alert)
- finished (without crashing), but not 100% successful (data not fetched)
- finished successfully
- no successful completion (regardless of number of attempts) within X time
- explicit error occurred
The only (weak) argument against this approach that I see (and this is an argument for putting this as a configuration option) - is that the reason I started using HealthChecks.io is because it's incredibly simple to set up for a cron job. Moving this logic to the client means slightly more complicated error handling logic to call the right endpoint for which type of failure.
The counter-argument is by the time you move from calling just "/success" to calling multiple endpoints, you're already in that position of more complicated client-side logic. If you want the simple "just run at least once every X hours" approach, then all you need to do is never call "fail" and set the grace period appropriately.
For our use-case, our logic for when to alert/not got much more complicated than described so the move to doing the rules in our code still made sense, but I think there are some other instances where we'd benefit from your proposal.
Its written that its 100 per metric scope, but I don't know what that means really.(2)
Also there seems to be no status monitor page ?
Web console allows check intervals of 1, 5, 10, or 15 minutes.
Sockets are a transport layer feature e.g. TCP or UDP. ICMP works at the network layer and has no notion of sockets.
Some systems (Linux, Darwin) also implement a special icmp socket type which can allow unprivileged ping.
Are you just trying to point out the misnomer?
I ended up writing my own service[0] to detect problems with graphql responses, before expanding it to cover websites and web apps too.
-[0]: https://onlineornot.com
I assume the rational is to not leak information about what's private. But still, it's weird.
I think either approach is valid as long as you're consistent. You can make a case for either 404 or 403 when you don't have enough permissions. In GitHub's case you can argue that it's a 404 because the resource does indeed not exist through your auth context. In AWS' case you can argue that a 403 makes sense because you don't have permission to know the answer to your query.
I sort of almost made myself feel a bit better about it by thinking 'no, it's not REST, we have reached the graphql server successfully and got a .. "successful" response from it, it's sort of a "Layer 8" on top of HTTP'. The problem is that none of the bloody tooling is 'Layer 8', so you end up in browser dev tools with all these 200 responses and no idea which ones are errorful. If any.
Especially because error responses from your web server layer are usually really different than errors from your backends.
The standard 503 and 404 codes?
Similar to how there's no difference between "there's no /pets/" or "there is no /pet/15", they are both 404 for /pets/15
400 is also a good code for "your query doesn't make sense". Anything but 200 really.
0 - http://spec.graphql.org/October2021/#sec-Executing-Requests
That's a WebDAV status code.
> GraphQL looks like it's been implemented by someone who thought 200 and 404 were the only possible codes.
Maybe. Or maybe they decided that a 2xx status would be interpreted as "success" by a non-trivial set of libraries and/or systems. Either way, take it up with the standards committee :-).
That doesn’t mean it’s not bloody annoying.
Those things are nice when you have a bazillion users and downtime means hordes of unhappy users and dollars flushing away at insane rates, but for the vast majority of hobby projects and even mid stage startups, what is described in this article is plenty good enough.
It's integrated with Mattermost currently, seems to work pretty well. Could also set it up on another VPS, for example on Hetzner (which also has excellent pricing), could also integrate another alerting method such as sending e-mails, or anything else that's supported out of the box: https://github.com/louislam/uptime-kuma/issues/284
Oh, also Zabbix for the servers themselves. Honestly, if things are as simple to setup as nowadays and you have about 50 EUR per year per node that you want (1 is usually enough, 2 is better from a redundancy standpoint, since then it becomes feasible to monitor the monitoring, others might go for 3 nodes for important things etc.), you don't even need to look for cloud services or complex systems out there.
Of course, if someone knows of some affordable options for cloud services, feel free to share!
I briefly checked the prices for a few and most of them are a little bit more expensive than just getting a VPS, setting up sshd to only use key based auth, throwing Let's Encrypt in front of the web UI (or maybe additional auth, or making it accessible only through VPN, whatever you want), adding fail2ban and unattended updates, and doing some other basic configuration that you probably have automated anyways.
The good news is that if you prefer cloud services and would rather have that piece of your setup be someone else's problem, they're not even an order of magnitude off in most cases - though I'm yet to see how Uptime Kuma in particular scales once I'll get to 100 endpoints. Seems like at a certain scale it's a bit cheaper to run your own monitoring, but at that point you might still find it easier to just pay a vendor.
At the end of the day, there's lots of great options out there, both cloud based and self-hosted, whichever is your personal preference.
I guess I'd personally also mention Contabo as an affordable host in general (though their web UI is antiquated), especially their storage nodes: https://contabo.com/en/storage-vps/
For the most part, though, use whichever host you've been with for a few years (though feel free to experiment with whatever new platforms catch your eye), but ideally still have local backups for everything (as long as you don't have to deal with regulations that'd make it not possible) so you can migrate elsewhere.
[0]: https://cronitor.io
* an uptime checker, in a container, running against however many services or sites
* uptime robot or whatever, against the local uptime checker
I think it covers quite a lot of things - the servers are up, DNS is OK, assets are OK. It can also be a safety net in case of other, more sophisticated monitoring fails to detect an unusual state.
This doesn't work well for website with too much javascript, ads or widgets.
SeleniumBase has a feature that looks at markup and which stores screenshots for visual comparison to a set baseline: https://seleniumbase.io/examples/visual_testing/ReadMe/
Found this blog describing use of SeleniumBase for that and OpenCV for screenshot comparison as another layer: https://blog.streamlit.io/testing-streamlit-apps-using-selen...
Also, the Holly grail of monitoring is to be able to remediate the problem automatically - this is pretty hard when users are reporting it.
Hacker mentality never left this site since inception :)
Full disclosure: my friends business
Hopefully if the service is worth reading, your customers will go and politely inform you. It is nice to have your work acknowledged.
The reality however is that every managed node (like on GKE) uses quite a lot of CPU and memory out of the box, for which the user pays. On top of that there're cluster fees, just for having it around. This makes it completely unfriendly to hobbyist projects, unless one is ready to pay dozens of $s just to have Kubernetes (prior to deploying any apps to it).
(And sure, there're free tiers here and there, but they never solve this problem completely on any of the big cloud providers, at least)
Compare that to managed "serverless" offerings (even pseudo-compatible with K8s API like Cloud Run), which eliminate the management fees, but impose a tax with latency. Oh well.
Compare this to a serverless offering that is multitenant by design, the control plane is shared making the overhead cost of an extra user is basically zero, which is why they don't charge you a fee like this.
IMO if you're a hobbyist interested in K8s, your best way to go is to install K3s, which is a lightweight, API compatible K8s alternative that runs on a single node. It's pretty nice if you don't care about fault tolerance or High Availability.
At the end of the day, all resources are run by the cloud provider on KVMs sharing the same physical machines anyways, so it's up to them how much to charge. The fact that both small and large customers get to pay for the same amount of resources allocated for them, only means these resources are not allocated in the most efficient manner. So a cloud provider could fix this.
We should also not discount the net positive effect of attracting more hobbyists and startups to your platform. That's how AWS and GCP started, for example, but now they're just focusing on more enterprise business so smaller ones mean less to them (although AWS arguably less so). But we shouldn't forget that while they don't contribute as much to the revenue, they're essentially a free advertising resource that make your platform stay "relevant" (and especially more so for burgeoning startups that could grow to bring more revenue in the future!). The moment they leave, the platform just becomes another IBM that's bound to die, for better or worse.
On top of that, the anti-analogy with serverless for control plane breaks down, because one could always run it on the same shared pool of resources in gVisor or Firecracker, just like with serverless.
Kubernetes probes are much different in my opinion.
Your Kubernetes liveness check will check if things are working inside of your cluster which is great for a high frequency checkup to potentially modify the state of your pod based on the result.
But Uptime Robot is an end to end test. It tests a real connection over the internet to your domain which exercises external DNS, traffic flowing through any reverse proxies, your SSL certificate, etc..
Both compliment each other for different use cases.
Fortunately using an email address when registering a new Let's Encrypt cert will let you get warned by LE if something happens where your cert doesn't get renewed and is about to expire.