Hacker News Reader

Hacker News

Comment by fhoffa | Hacker News Reader

Parent Full thread

fhoffa

From the Twitter Hadoop to GCP talk that Boulos linked:

- >500k cores

- >300PB storage

- >12,500 cluster size

- >1T messages per day

And there's also this other talk, "How Twitter Migrated its On-Prem Analytics to Google Cloud" - focused on their migration to BigQuery:

- https://www.youtube.com/watch?v=sitnQxyejUg

- 20 TB/ day of raw log data, >100k events/sec

- Loading ~1TB/hour into BigQuery.

- Serving 5,000+ complex queries / second. p99 ~300ms

Disclosure: I'm Felipe Hoffa and I work for Google Cloud https://twitter.com/felipehoffa.

View on HN

At this scale I'm a little surprised Twitter saved money on the migration. My assumptions were that at this scale the upfront cost of migration trumped any kind of difference in annual maintenance, esp. considering you still do need an SRE team anyway who knows big data well enough to solve any and all software related problems

At that scale the team itself likely costs peanuts.

yeah but you still need the IT team too. Not sure, from my conversations at cough cough other media giants the economics of moving to cloud don't always work out at massive scale

Oh, certainly. I was talking more about the fact that the monthly cost of 125.000 instances likely beats the cost of your IT team whatever it’s size is.

I’d be interested in their cost/benefit analysis for this one.

Yeah, what I mean is it's not just cloud costs vs. local hosting costs. You have to come up with different processes for SRE and IT (removing mundane hardware work, accounting for cloud idiosyncrasies), benchmark, make sure your hosting cloud can handle scaling up to some upper bound estimate of your future user base, etc. At a company the size of twitter these become legitimate concerns. I've been throttled on certain services of certain cloud providers at 600 data nodes lol... of course twitter can negotiate higher allowances but massive scaling doesn't happen with the flick of a switch, sometimes you need big, long-term, expensive contracts

12,500, not 125,000. Still a lot ;)

Automation, the twitter team is full of SV veterans who have run into many problems at scale years ago. At least it used to be.

> 1T messages per day

How? I mean there are 7B people on earth. Are people that active on Twitter? Or does "messages" mean something else here?

There are tons of bots on Twitter

Bots, news organizations, and teenagers. ~143 messages/day per human doesn't seem unreasonable.

A message in this case probably doesn't mean a Tweet. A message if probably just a message on the bus/in the queue and any action on Twitter probably produces one. Like, retweet, tweet, follow, sign up, everything.

Oh if the likes are included then it makes sense.

A message probably counts every tweet a person views at a minimum.

It's stunning how many resources we as a species are wasting on something of so little net positive value to humanity as Twitter.

I'm not a fan of or user of Twitter, but I can't deny that anything that allows people to communicate instantly, easily, and broadly is a benefit to humanity.

So in your opinion Facebook is a benefit to humanity also? Which conflicts with nearly every study of it ever done?

Reply on news.ycombinator.com