Comment by jasode | Hacker News Reader

Comment by jasode | Hacker News Reader

Parent Full thread

>Facebook is kind of an interesting example, as they got pretty far into "hyperscale" with mostly PHP, memcached, and Mysql.

But PHP and MySql are only the surface level tools we see.

There is also the unseen and that's where the immense complexity lies hidden under the surface.

E.g. To help with scaling (which Friendster failed at), Facebook had separate MySql instances for each school. An interview with Facebook co-founder Dustin Moskovitz mentioned they had constant stress of emergencies and fires to put out. They had tons of custom tooling, Linux scripts, batch jobs, automation/orchestration/dashboard, etc and it's very likely the combination of all that ends up inventing an "invisible internal Kubernetes" without calling it "Kubernetes" to manage their server fleet. In other words, if Docker/Kubernetes/AWS were around, the early Facebook team might have offloaded some of the complexity onto a 3rd-party to reduce the all-nighters. But the tools didn't exist yet so they had to do it the hard way and invent "deployment orchestration" themselves.

[EDIT to reply]

Thanks for the "Tupperware" comment. A Google search leads to a Facebook post saying they renamed it to "Twine":

https://engineering.fb.com/2019/06/06/data-center-engineerin...

Facebook released it's PHP to C++ compiler [0] in 2010.

Maybe it was used internally before that. So they got pretty far from "normal PHP" relatively early.

[0] https://en.wikipedia.org/wiki/HipHop_for_PHP

That's recent though. I'm saying they scaled up quite a long way before doing anything like that. From the outside it looks like they didn't really branch out into things like this until 2008 or so.

>From the outside it looks like they didn't really branch out into things like this until 2008 or so.

No, even before the Newsfeed feature rollout of September 2006, they already had crazy complexity orchestrating multiple servers for many schools. They were only a "simple PHP+MySql" website when it was February 2004 with only Harvard students on it. Dustin said he was the one in charge of spinning up new servers for each new school and he said it was a nightmare of engineering effort.

IIRC, they were doing things like patching bittorrent to prefer closer IP's to pull from during their deploys. So it would roll through their network from 1 side to the other without just completely saturating the entire network.

But the point still remains, they had challenges to solve, but they were also simple, which made it possible TO solve those challenges.

But today, 16 years of Moore's law later, you could now have gone way, way, way, way past that.

They probably could have done most of America on a single server.

For example, even in 2009 Stack Overflow was running off two dedicated servers:

https://stackoverflow.blog/2009/01/12/new-stack-overflow-ser...

Stack Overflow in 2009 is an example of something that's cacheable extremely well, several orders of magnitude than stuff like personalized walls for every user.

it's called 'tupperware'

Reply on news.ycombinator.com