How We Manage a Million Push Notifications an Hour

The one-day record could have been serviced in 5.8 seconds of peak per-second throughput, is that right?

I really like the diagram in this blog post. Is that diagram custom made or did you use software to create it?

ROFL : I was about to post a message here bragging how i achieved 2 millions push a minute with a simple kafka queues + a cluster of push servers running on go architecture, but now i feel humbled !

No offence but is this correct?

> We recently hit a peak of 850 Million notifications per second

per second?

From your blog "OnePush is fast - We've observed sustained deliveries up to 125,000/second and spikes up to 175,000/second."

I think you may have a typo. The bandwidth would be incredible too, if it were an unlikely 10 bytes per delivery that would be 8.5 GB/Sec.

I wrote an algorithmic trading framework to work on Warsaw Stock Exchange.

WSE back then had a limit of 10k messages per second (as in one message every 1/10000th of a second). Messages came by two separate network operators so that was 20k messages to be deduplicated to 10k operations. Incomming messages came in compressed so they required uncompressing.

Responding to the message required complex processing and then may have resulted in an order to market which, again, had to be constructed and validated.

All this worked on a single server (regular two-socket Xeon-based server).

There was 10us (microseconds..) time budget to send response to the market and it had to work every time even under maximum possible load (10k messages per second).

Routing and forwarding 300 messages per second doesn't seem like something to brag about...

And what happens if the raspberry pi goes down for an hour ?

I think this is a disingenuous or naive view, while you may not have been serious, I think it's worth unpacking why it's naive as I think a lot of engineers have similar opinions.

To start, there's nothing here about what machine this architecture runs on, it could be running on a Raspberry Pi for all we know.

Then, this ignores the cost of database lookups for the keys. That data is probably small enough to be on the one machine, but then you have to have service support for (reliably, in real time) syncing that data to the service. A separate database is therefore probably the right solution here, which means you're doing networking in each message send, which makes it unlikely that a Pi could do this.

Next up you've got the issues of reliability. The message queue separation gives you better reliability in the face of issues such as upstream APIs going down or erroring, or issues for a specific user. All the business logic around handling this, the message queue handling persistence and ACID semantics (or parts of it), this all takes additional resources, not to mention potentially a fair bit of disk space (for a Pi) to queue up undelivered messages should an upstream API slow down or stop accepting new messages.

Then you have hardware failure, at this scale you don't want a single machine failure to wipe out your primary communication method with millions of customers. You'd therefore want to have a distributed system, even if that's only for reliability rather than performance.

Lastly, 21 million clock cycles might sound like a lot, and might go a long way with C/C++/Rust, but as you move up to more dynamic languages that will reduce significantly. It happens that they are using Go here, and that's likely to get pretty good performance out of the hardware, but writing this service in Python/Ruby would be a very valid choice for developer productivity, or based on existing skills they have in the team. That might be 1/10th the performance, but since you need a distributed system for reliability here anyway, adding a few more machines to the pool might be a better choice than introducing a lower level language that takes longer to develop and exposes you to memory safety or threading issues.

There may well be other factors I haven't considered here, but I think for the use case of delivering that scale of messages, the reliability options you get with a system like this are well worth the additional hardware requirements and architecture overhead.

Edit: lmilcin makes a good point about trading systems, but there are several differences – that system still has a single point of failure, it was probably written in a low level language with input from experts on performance, and it was running on a much faster machine. The single point of failure of a server-grade machine like that is probably an acceptable risk if you own the hardware, but in a cloud environment (which brings other benefits) hardware is less reliable so probably not an acceptable risk there. I don't think it's an apples-to-apples comparison, although it is interesting.

Yep, I hit the calculator first thing too, but I think your criticism is not quite fair, after all the data has to be pulled off disk and acknowledgements written back.

That said, it really doesn't sound too difficult with straightforward architecting (says the armchair critic).

My thoughts exactly.

One million per hour? I forgot how to count that low.

What if you get one push event on the first second, and then 999,999 on the last second? Still one million per hour. Don’t assume the events are evenly distributed. I see this a lot on HN. Yes, 1,000,000 divided by 3600 is indeed about 278, but why would we assume it was 278 notifications exactly every second?

Same reaction. You could handle this kind of traffic with, just as examples, procmail or inetd on a single PC.

> but id guess theres a good reason.

Sometimes there are no good reasons, and we should have the ability to question it sometimes, that's how progress is made, in science and in engineering.

Why not use Amazon SNS? They abstract away the differences between APNS and whatever Google are calling their push infrastructure these days.

Did you read the post? They're using Firebase messaging, but that only covers a part of their workflow.

Pretty soon your app is just a bunch of proprietary cloud provider products glued together and now there’s vendor lock-in and much less flexibility.

You might want to throttle messages based on user settings or say ML models to avoid too many annoying messages. For critical stuff you might want to switch over to email or SMS.

These are startups that have the funding to do this kind of thing. This is a core part of the stack for any such company.

One big factor is being free from bundling proprietary, vendor-specific libraries that may also leak user's information or otherwise increase the security risk footprint of your app. Having control over the services that make up the core of your business, as opposed to outsourcing and relying on 3rd-parties, can be very freeing and also cost effective in the long run.

I cofounded OneSignal for this reason. We were previously a game studio that built an in-house solution. It was a huge maintenance headache and a distraction. Eventually, we realized that so many companies were one-off push implementations and decided to focus our efforts on building a really great third-party solution.

Firebase and SNS have since entered this market as well. I'm obviously biased, but I honestly can't recommend them. They work for some of the basics, but there's been very little innovation in either product and occasional service issues that go unsolved for a long time. They also don't make any money from them, which should be a concern for anyone that plans to use these services for such a core part of their business.

It's not hard to do what they did, so I'm not surprised they did it themselves. It's just simple async queue processing, and doesn't really need a lot of sophistication, especially at just 1M notifications per hour. Now if they need to process a few orders of magnitude more, then AWS SNS would probably be a good next step.

Gojek is a super app in Indonesia serving a population of 264 million. They can afford this level of engineering and are required to do it, to stay competitive. They have millions of motorcycle driver (called gojeks) for whom they are trying to find jobs while idling waiting for passengers to transport.

If uber built these no one will question them why don't they use some off the shelf solution. Gojek is a google cloud customer and under their startup program - surge. So I am sure they are aware of the solution proposed.

They don't call themselves technology company but company which helps gojek drivers to find jobs and earn extra. In the process they became a food delivery, transportation, payments, courier company. In spite of this they still focus on their primary objective to find jobs for gojek and let them earn extra for better living.

Before Gojek app most of the drivers were just idling waiting for passengers, now with a $25 Kai OS feature phone [1], they can just receive a message and earn instead of just idling, they don't have to use iPhone or Android or some heavy OS. Indeed based on social survey some of the drivers could increase there earning 3-4 times based on how willing are they to take up additional jobs.

I believe the beauty of Gojek lies in its model to focus on finding jobs and developing solution to help those motorcycle drivers using technology. Although they extended it to cars. I am intrigued by their idea since its not a copy of Uber, but a solution to a problem local to Indonesia, which can extend to any country in the world finding jobs to do while idling and earn more.

[1] https://en.wikipedia.org/wiki/KaiOS

Engineering Pride (and exercise for future employment).

While this might sound condescending but hear me out:

- if you keep doing the same thing like everybody else out there, gluing APIs from cloud-vendor, you won't stand out.

- if you're a company of _that_ size (Uber, Gojek), you gotta do something big because the name/brand is associated with "Engineering" excellence

- if you're a leader (or in management), would you tell your employee to "suck it up" and do the boring stuff like everybody else? or would you empower them to do more? hype them up? this is a strategy for keeping them happy too.

I don't begrudge that. That's how companies evolve: they start small and scrappy, they grow very fast, and they're at the level where "doing simple stuff" won't get you promotion or you won't have something cool to show during the interview for your next job.

It's work dynamic.

``` Surely there are other more important features delivery business value to be done rather than re-inventing (and then scaling and supporting) the delivery of mobile notifications. ``` the business value the service provided is that this allow the engineering organization to scale. Gojek has couple hundred engineers, split across multiple teams, where each teams own multiple services/products/applications, each with their own development cycle. Having a single abstraction provided as a service helps to mitigate the risks involved in changes to implementation details and external dependencies (e.g. GCM deprecation) since it abstracts it away completely.

Go-Jek has been huge for us as holiday rental managers in Bali. But lately there seems to be some sort of racket by drivers:

Example for Go-send (sending a package): "I have motorbike problem please can you cancel".

A minute after cancelling I get a message on WhatsApp: "Hey fixed my motorbike, still need?". Same thing with food orders.

Example for the Go-food: "Hey I'm at the restaurant, the phone is down. Can you cancel order? I will do manually".

Obviously the restaurant and the driver gets the full customer payment this way. Go-jek gets zero. This depends on the customer cancelling as otherwise the drivers rating takes a hit.

Its difficult to "guess" ahead of the fact how you are going to scale for the next 10x, because you need to know.

a.) Which component is going to start breaking, and that depends on the usage pattern.

b.) Which business/tech compromises are ok to make, and that depends on a.)

Generally speaking though, you'd try to benchmark the system to find the bottleneck component, and based on the nature of it, either try to throw more hardware (horizontal or vertical scaling) or optimize the software.

We handle peaks of 800k tps in few systems. It is for an analytical platform. Partition in Kafka by some evenly distributed key, Create simple apps that read from a partition and process it, commit offset. Avoid communication between process/threads. Repartition using kafka only. For some cases we had to implement sampling wherein usecase required highly skewed partitions.