Handling five billion sessions a day in real time | Hacker News Reader

Handling five billion sessions a day in real time

blog.twitter.com

98 pointsknes22 comments

What i never see is how much servers/hardware/resources they are spending. It's _always_ like "we process a quadrillion messages per second" but it never mentions that you'll need a quadrillion expensive servers to do so. Same for many many database benchmarks, sadly...

It's a couple years old but What it takes to run Stack Overflow[1] lists the number of DB servers, Web servers, etc

[1] http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-s...

This update is more recent:

http://highscalability.com/blog/2014/7/21/stackoverflow-upda...

StackExchange is pretty open with their hardware metrics.

The StackExchange performance site: https://stackexchange.com/performance

> RAM: 48GB

Per server or in total..? Either one is believable.

I don't think a single server can do its task in just 5 GiB (48 GiB/9)

It must be per server, adding RAM is really the quickest and cheapest way to improve performance.

See the tagline "statistics by server"

Image: http://i.imgur.com/q1oDamy.png

I had a hunch I'd see the data structures du jour in there: Bloom Filters and HyperLogLog. Since they are pretty accurate by themselves already but run much faster than the brute force approach, why keep the latter? The error margin, if used properly, should be small enough so people shouldn't really care.

They are not for accuracy. One usage for instance is to recompute stats if there are bugs in the stats processing code. Or if you want to add more metrics. (I work at Twitter but not on Answers).

> What happens if our speed (real-time) processing layer goes down? Our on-call engineers will get paged and address the problem.

Innovative!

But for real, that's some architecture, and very insightful to see what they've got. Totally nothing I'd have guessed.

First thing I noticed was Amazon in there Stack. Amazon really owns the internet...

In what way does Amazon own the Internet?

Yes, they own some of the computers that make up the Internet, but by far not all of them.

they are the dominant vendor of hardware/hosting to the largest and most complex web applications on the internet. AWS is the pillar that some of the most high volume sites on the internet (like Twitter and Netflix) rely on. Its safe to say that without AWS (or some other similar vendor) the internet as we know it wouldn't exist.

That's ridiculous. It's like you're saying: without Honda, the car industry, as we know it, wouldn't exist.

The hosting industry is huge. There's a ton options, many of them better, larger and older than AWS. There are so many sites that don't use AWS. For example, I believe that, pre-Google, youtube used ServerBeach.

> There's a ton options, many of them better, larger and older than AWS.

Some examples would be more interesting than a general "you're wrong".

For every site that's reliant on AWS, there's probably more than 10 000 that aren't. The giants, for example: Google. Facebook. Wikipedia. Microsoft.

It's hard to get actual numbers, but Equinix might be the largest. I believe their Singapore data center is actually what AWS, Digital Ocean and Azure use. (there's a lot more of these, such as Internap and Level 3 (which provided (and might still) services for Netflix and iTunes.))

AT&T is quite large. World of Warcraft used to be hosted on AT&T's infrastructure (might still be).

The above tend to be for large setups. Going smaller, you have the Rackspaces and Softlayers (now IBM). And, to a lesser extent, providers like OVH and Hetzner.

And, you can't forget the likes of Digital Ocean and Linode.

It's hard to come up with examples because more things are examples than aren't. AWS' market share is growing, but that's to be expected, in a lot of ways, despite EC2 being ~10 years old , it's still a new player.

AFAIC, what's more interesting about AWS is:

1 - How dominating it is over Microsoft and Google

2 - How popular the less invasive offerings are. A lot of people are sticking to their colocated, dedicated or VPSs, while using some part of AWS for DNS, backups or email. I hope they keep this up.

Thanks for the reply—exactly what I was hoping for!

I'm not saying right or wrong but here are some examples:

WhatsApp (Pre-Facebook) - Softlayer

Youtube (Pre-Google) - ServerBeach

Vevo - Rackspace

Zendesk - Rackspace

GoCardless - Softlayer

Snapchat - Softlayer

twitter: @copypastaa and then appending ?hn is at least consistent :)

"How we handle spying on your mobile phone activity on a global scale"

hahaha but it is a good architecture you have to admint :D

Reply on news.ycombinator.com