“just put Twitter in the cloud” we tried that- $300M monthly | Hacker News Reader

“just put Twitter in the cloud” we tried that- $300M monthly

twitter.com

63 pointsren_engineer28 comments

I can’t understand how cloud means “simple and cheap” to so many people and companies. In fact it’s the absolute opposite.

There’s tons of much, much more powerful and cheap servers you can get online than those of the major clouds, often with little or no traffic fees.

Most companies are too lazy to even investigate the choices.

More than laziness, it's too complex a problem for most decision makers. Environment is ever changing and extremely competitive. Most people don't handle such problems well and few get the training or have the support. It's pretty much like crypto. Most people are sold shit they have no capacity to understand. So the story writes itself.

These are info asymmetry problems and society and business hasn't been dealing with them at such scales for too long. So solutions are a work in progress.

The cloud is great when you are starting out, but at scale you are going to hemorrhage money. This is why it's dangerous to tie yourself to all those cloud "toys" and then rapidly expand.

If I just want to have my own little home page, a micro EC2 instance will do just fine - I don't want to go to a colocation for that.

Once you are at scale, you need to get out of the cloud. I agree with DHH on this.

https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47...

Often people will say “AWS is so much bigger than us, they have to accommodate us / what could we know that they don’t” is that AWS is still building for the median customer (+/-). It doesn’t make financial sense to optimize for the extremes and extremes can be extreme (per the tweet thread). If you’re a reasonably technical company there’s a good chance something about your main workloads will challenge AWS.

I wouldn't call my company reasonably technical and we frequently hit barriers in AWS. Twitter has no chance on AWS.

deployments alone are a challenge. there are so many race conditions it’s sometimes impossible to do it reliably.

300m/month to deliver text, some photos and a bit of video to (admittedly tons of) people sounds insane.

Then again based on the architecture sketch Elon posted https://mobile.twitter.com/elonmusk/status/15938990295318036... most of the compute is being used by ... the feed recommender AI crap that works awfully anyway? Insanely bad job programming this product.

Developing for the cloud makes sense when you're developing features as fast as you can leveraging cloud services to make it all easier. When you have a relatively stable feature set and scaling massively it's better to use private datacenters.

Snapchat pays Google cloud $400MM a year. Using these cost metrics, are Twitter's compute needs 9X those of Snapchat? maybe so. seems reasonable to me. And then perhaps, this person is saying that cloud (Google or otherwise) is not the way to go for Snapchat

https://www.vox.com/2017/3/1/14661126/snap-snapchat-ipo-spen...

The cloud is for peace of mind (in some ways)... not for saving money

>500,000 m6gd.12xlarge systems

And I think HN still runs on one server? 500,000 for twitter seems a tad inefficient

Did you compare HN with Twitter?

lol, right?

I guess kinda. Googling HN has about 300k daily active users, Twitter about 250m so 800x approx.

Do you really think the number of users tells the whole story? The websites are very different.

“in the cloud” - is it a desktop app now? or is the world cloud synonymous with aws, and aws only?

Isn't this sorta what Oxide Computer is trying to solve? Where you can have your cloud and eat it too?

I guess this makes FTX's $250K a month spend on AWS downright thrifty.

there are banks that also spend millions per month on AWS

probably the same banks to say no to vendors who cannot provide on-prem solutions. but, but, but it's ok if you're AWS.

or if you say "on-prem is on the roadmap"

m6.12xl are 48 vCPU systems. 25 million cores is a hilariously huge amount of compute to use to run a site of twitter size, which I guess does point to how much tech debt they have ignored over time.

They should have probably written it in Assembly.

By "twitter size" are you implying it's... small?

Twitter is the 4th most visited site in the world.

https://en.wikipedia.org/wiki/List_of_most_visited_websites

But it’s basically just simple data moving, it should be rather trivial in terms of compute power needed.

There is nothing simple in data moving. Problem is that it can be abstracted away so you do not have to worry about it. Out of sight out of mind. Example - majority of the CPU area is devoted to data moving problem (cache, branch prediction, out of order execution, register renaming). The same is happening on other levels of the stack (OS, network and so on).

It sounds high for sure.

But they do serve videos, which possibly they have to transcode or otherwise process on import (?) They probably also have some filters for pictures to remove porn and stuff like that.

This can amount to serious CPU if you have millions of users uploading that stuff daily.

A pure text-based Twitter would be much smaller.

There would be better EC2 instances (or AWS services) for that though.

Reply on news.ycombinator.com