I'm curious why/if this architecture was chosen. I get that it started as a standalone product (Amixr), but in the current state it is hard to rationalize deploying this next to Grafana in my current containerless setting.
Helm (https://github.com/grafana/oncall/tree/dev/helm/oncall), docker-composes for hobby and dev environments.
Besides deployment, there are two main priorities for OnCall architecture: 1) It should be as "default" as possible. No fancy tech, no hacking around 2) It should deliver notifications no matter what.
We chose the most "boring" (no offense Django community, that's a great quality for a framework) stack we know well: Django, Rabbit, Celery, MySQL, Redis. It's mature, reliable, and allows us to build a message bus-based pipeline with reliable and predictable migrations.
It's important for such a tool to be based on message bus because it should have no single point of failure. If worker will die, the other will pick up the task and deliver alert. If Slack will go down, you won't loose your data. It will continue delivering to other destinations and will deliver to Slack once it's up.
The architecture you see in the repo was live for 3+ years now. We were able to perform a few hundreds of data migrations without downtimes, had no major downtimes or data loss. So I'm pretty happy with this choice.
Sorry but why is rabbitmq really necessary?
To be fair, even in its current form, it should be possible to operate this system with sqlite (i.e. no db server) and in-process celery workers (i.e. no rabbit MQ) if configured correctly, assuming they're not using MySQL-specific features in the app.
Using a message bus, a persistent data store behind a SQL interface, and a caching layer are all good design choices. I think the OP's concern is less with your particular implementations, and more with the principle of preventing operators from bringing their own preferred implementation of those interfaces to the table.
They mentioned that it makes sense because you were a standalone product, so stack portability was less of a concern. But as FOSS, you're opening yourself up to different standards on portability.
It requires some work on the maintainer to make the application tolerant to different fulfillments of the same interfaces. But it's good work. It usually results in cleaner separation of concerns between application logic and caching/message bus/persistence logic, for one. It also allows your app to serve a wider audience: for example, those who are locked-in to using Postgres/Kafka/Memcached.
One of them allows a thousand different nodes on different networks to share a single dataset with high availability. The other can't share data with any other application, doesn't have high availability, is constrained by the resources of the executing application node, has obvious performance limits, limited functionality, no commercial support, etc etc.
And we're talking about a product that's intended for dealing with on-call alerts. The entire point is to alert when things are crashing, so you would want it to be highly available. As in, running on more than one node.
I know the HN hipsters are all gung-ho for SQLite, but let's try to reign in the hype train.
This is for tiny shops with 4 servers. And tiny shops with 4 servers don't have time to spin up a horrendous stack like this. I was excited to see this announcement until I saw all the moving pieces. No thanks!
At the very least, SQLite should be the default database for this product, and users can swap it out with their MySQL database cluster if they really are Google-scale.
An important question to ask is how much availability are you actually gaining from the setup. It wouldn't be the first time I see a system moving from single-node to multinode and being less available than before due to the extra complexity and moving pieces.
They chose this stack, it works for them. They’ve put it through its paces in production.
It’s as boring as it gets.
I get why you would add a relational DB to the mix. Personally, I'd like a Rabbit-free option.
https://prometheus.io/docs/introduction/overview/#architectu...
When comparing to OnCall you need OnCall AND still the rest of that Prometheus picture.
Compare with this picture where everything in the leftmost Alert Detection box is what you see in the Prometheus architecture. https://grafana.com/docs/oncall/latest/getting-started/
MySQL is a weird if not slightly disturbing choice. Other than that it's a boring, battle-tested stack that is relatively easy to scale. I agree that Go is nicer, but I'm biased by several years of dealing with horrific Flask / Django projects.
I think you misspelled "beautiful"
Complexity comes at a steep price when something critical (e.g. OnCall) breaks and you have to debug it in a hurry.
Shoving everything in a container and closing the lid does not help.
IMHO, OP just stated that one could solve this with less dependencies and have the same (if not a better) result.
IMHO a fat binary written from scratch would have been a way worse choice than to use a standard stack, both in terms of bugs and time, let alone Open Source contributions or any scalability.
In terms of number of services, what do you get rid of that produce a better result? maybe RMQ and use a worse queue?, celery and write your own task manager or use another dependency?
Far less scalable, but it is dramatically simpler to deploy. Often gets you surprisingly far though. Would be interesting to know how many monitored integrations could be supported by that flow.
Not to mention that having multiple components doesn't mean it's "scalable" by default, it could happen that some part of the pipeline doesn't like multiple instances of something.
Doesn't have all the plumbing you'd want, there is a wrapper (https://github.com/bretth/django-pq/) that seems to give you an entrypoint command more like `celery worker ...` but I've not investigated it closely.
A simple way to use an RDBMS as a message queue, that has been in use since before most HN readers were born, is roughly:
- enqueue an item by inserting a row into a table with a status of QUEUED
- use a SELECT FOR UPDATE, or UPDATE...LIMIT 1, or similar, to atomically claim and return the first status=QUEUED item, while setting its status to RUNNING (setting a timestamp is also recommended)
- when the work is complete, update the status to DONE
The first software company I worked for was using this basic approach to queue outbound emails (and phone and fax... it was 2005!), millions per day, on an Oracle DB that also ran the entire rest of the business. It's not hard.
Python(Django)+Redis+[SQLite]
Python(Django)+Postgres
[Compiled Go binary]+[SQLite]
SQLite barely even counts as an architectural dependency TBH :)
Just think about Gitea vs GitLab.
As for python, at least getting a dockerfile helps a lot. Otherwise it's a huge mess to get running, yes.
Python is still a hassle anyways, since the lack of true multithreading means that you often need multiple deployments, which the Celery usage here for instance shows.
Maybe I'm behind the times, but I can't figure out what you mean here. As far as I know 'java -jar' or servlets are still the most common ways of running a Java app. Are you talking graal and native image?
But using jlink for java, one can package everything to a smaller runtime distributed together with the application. So then I feel it will be not much different than a Go executable.
> The generated JRE with your sample application does not have any other dependencies...
> You can distribute your application bundled with the custom runtime in custom-runtime. It includes your application.
From the guide here https://access.redhat.com/documentation/en-us/openjdk/11/htm...
Maintaining tenths of binaries pulled from random github projects over the years is a nightmare.
(Not to mention all the issues around supply chain management, licensing issues, homecalling and so on)
Additionally, distribution packages are tested by a significant number of users before the release.
Nothing of this sort happens around any language-specific package manager. You just get whatever happens to be around all software forges.
Unsurprisingly, there has been many serious supply chain attacks in the last 5 years. None of which affected the usual big distros.
MVS also prevents unexpected upgrades just because someone deleted a lockfile.
I guess we can argue about "big" but didn't both Arch (https://lists.archlinux.org/pipermail/aur-general/2018-July/...) and Gentoo (https://wiki.gentoo.org/wiki/Project:Infrastructure/Incident... and older, https://bugs.gentoo.org/show_bug.cgi?id=323691) have actual compromised packages? And also not five years ago, but Fedora (https://lists.fedoraproject.org/pipermail/announce/2011-Janu...) and Debian (https://www.debian.org/News/2003/20031202) had compromises but no known package changes.
What has happened in the package ecosystem to make you believe this? Is it velocity of updates or actual trust?
I haven’t heard of any malicious package maintainers.
For other kinds of quality, I have my own tests which are much more relevant to my use cases than whatever the distro maintainers are doing.
I've been a DD and while distros do work to integrate disparate upstreams as well as possible, they rarely reject packages for being fundamentally low quality or make significant quality judgements qua their role as maintainer (only when they're a maintainer because they're also a direct user). Other distributions do even less than Debian.
In the end most distros will be saved by the fact they don't upgrade quickly. Which is also accomplished by MVS without putting another attack vector in the pipeline.
There's more than a hundred package maintainers (I'm not sure exactly how many), but the median is about 50 packages.
Do you think people can't keep up with the updates for 50 packages?
To paraphrase, all "more than a hundred" of those people need to be lucky every time.
It even ships in containers along with Docker Compose files and Helm charts, which would suit the deployment use cases of 99% of users. I understand that you're not using containers, but I don't think that's a limitation that many are inflicting upon themselves as of late, and if pressed, installing Docker Compose takes about 5 minutes and you don't have to think about it again.
Except when you need to pin and repin versions to comply with a security policy, which may be why you aren't even running containers in the first place
i love django projects but mysql, celery and rabbitmq -- no thanks.
Redis itself has had solid support for building reliable distributed task streaming for nearly 4 years (Redis ConsumerGroups introduced in 2018).