AppLovin: Processing 30B Requests A Day | Hacker News Reader

AppLovin: Processing 30B Requests A Day

highscalability.com

101 pointscoloneltcb17 comments

And I was proud of my API backend processing 70M requests a day... That's suddenly feeling like a lot less traffic and a lot less spectacular.

On what hardware was that?

AWS, ~11 EC2 c1-medium boxes (auto scales), one db.r3.4xlarge RDS MySQL server, plus a beefy Redis server (instance size escapes me).

FreeBSD 10 on the EC2 boxes, whatever Linux AWS is running on their managed service instances. Nginx backed by Python and Go servers.

Plus a smattering of other services like ELBs, etc.

I can't imagine having something so critical that you build a custom Nginx plugin. I thought I was cool for doing a bit of Lua scripting. :)

How were your monthly expenses like for this setup?

Although it's difficult to fully articulate, as our traffic is basically a bell curve through the day and we auto scale between say 7 and 13 servers throughout the day... We pay about $4200/month. That includes several TB of traffic (which we gzip compress where it makes sense).

I was able to process about 250M requests a day on a dedicated server (E3) without breaking a sweat.

It depends on what you're doing in those requests.

Ex: Realtime stats system. On each request doing several get,set,increment,push-into-queue, geoip + 1 bulk-postgresql-upsert every second. 1ms response, 400 requests/second, 3 python processes, 10$/month shared hosting.

Thanks for sharing this - it's great to see this level of detail on an adtech stack. I wanted to ask a little more about the user profiles in Aerospike. What process is writing 12k profiles a sec, and what's the read path - is Aerospike e.g. wired into Nginx using https://github.com/aerospike/aerospike-mod-lua ? Any gotchas with Aerospike versus e.g. a Cassandra or DynamoDB?

We don't really use Lua: we connect to Aerospike directly, using a C driver. This saves a lot of time and allows sub-1ms responses.

I find Aerospike latency a lot more stable compared to that of Cassandra. Our use case is quite simple: we treat Aerospike as a large and fast key-value storage.

Thank you for your input again! May I ask if you run Aerospike on an in-house cluster to reduce latency, or run on AWS?

Some amazingly cool stuff that any engineer would want to be part of! Could anyone answer a few questions?

1. Is the Spark cluster running on top of an in-house cluster or on AWS EMR/S3?

2. Any more details about the custom message system, specifically how to guarantee exactly once delivery?

3. How do you guarantee data is never dropped?

4. "Make sure every message can be re-played without data corruption" how to achieve this when there is so many data points coming in?

Again, thank you for the insights!

Thank you! Let me try to address those questions:

> 1. Is the Spark cluster running on top of an in-house cluster or on AWS EMR/S3?

It runs on an in-house cluster.

> 2. Any more details about the custom message system, specifically how to guarantee exactly once delivery?

This topic might deserve a separate article. We have built a custom locking system that ensures proper delivery.

> 3. How do you guarantee data is never dropped?

We write each data point into 2-3 locations in our system and also back it up in S3.

> 4. "Make sure every message can be re-played without data corruption" how to achieve this when there is so many data points coming in?

Basically, batch things and keep a lot of signatures and checksums around.

Thanks for your time!

About point 4, I'm assuming you built in idempotency into your message replay system. Is your unit of idempotency, say, 1 hour or some other unit of time, or is every single message have idempotent results? Because the latter would be pretty amazing :)

I remember seeing people complaining that their accounts on google play got banned due to security issues with AppLovin SDK. Example: http://forums.makingmoneywithandroid.com/advertising-network...

Any comment on that?

Hi kumarm - Michael from AppLovin here. Thanks for the comment, it's an important concern. Here is a blog post outlining the problem and corrective actions that were taken: http://blog.applovin.com/applovin-security-notice/

How do you loadbalance requests between the servers?

We use a combination of software (ipvs, haproxy) and hardware load balancers. We plan on gradually moving away from traditional hardware load balancers.

Reply on news.ycombinator.com