Hacker News Reader

Comment by jaclaz | Hacker News Reader

Side question.

How many/which companies have more than one million Linux machines?

At least Facebook (where OP worked), Amazon, Google, and Microsoft. Probably Netflix, maybe Apple. There might be a couple more, but no more than that because we've already accounted for a pretty high percentage of worldwide shipments for servers, disks, etc. Fun fact: when you're that big, your demand creates its own inflation and you have to consider that in projections.

Thanks, so a handful at most, and the "usual" ones, I always thought that those companies keep their machines connected in (redundant) "sets" and that a command affecting all of them was more a case for "never" rather than "once in a while".

Google, at least, has a thing that is supposed to prevent widespread disruption at the machine level, called the "Safe Removal Service"[1]. This is a good idea that in practice isn't perfect. If you write a tool that does not consult SRS, or your service doesn't declare a SRS policy, there can be surprises.

A particular outage that I will never forget took out Gmail delivery worldwide in an instant, because the change was not expected to be disruptive and therefore did not integrate with SRS. As it turned out the change disabled the machines where it was applied, and the process of selecting a subset of machines to canary the change was not independent of the way in which Gmail assigns services to machines, so in the space of a few seconds they created a global outage.

https://twitter.com/bgrant0607/status/1134536670504554496

If by "machine" we also mean things outside of a 19" rack, I would wager that large telecoms probably have way more devices running Linux than FAANG. Imagine the network of cable modems that Comcast alone must operate. What percentage of their 28+ million broadband customers rent Comcast owned/managed modems? Almost all of them except the tech-savvy crowd? And that's just one device type.

Not to mention the networks of cellular base stations worldwide that run extremely sophisticated systems (if not Linux itself).

The number blew me away. But does she mean in one location or VMs?

One million is a lot no matter how you slice it.

How do you define one location? If it's like, a contiguous plat of land with a bunch of buildings, each containing suites, and each of those containing clusters... then these days, yeah, that's probably not too much of a stretch.

And yeah, physical machines, not VMs. Sometimes they're blades, sometimes they're sleds, but I mean real hardware made out of metal that you can pick up and use to defend the datacenter if you have to.

(Although, honestly, I was talking about global counts in the million+ range when I wrote it since it was referencing the past, but by now, a region with a million+ is not far-fetched.)

Reply on news.ycombinator.com