Hacker News Reader

Comment by mtejo | Hacker News Reader

Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!

View on HN

Nice work! Does Twitter runs their own DCs or hosted somewhere else?

As of three years ago, Twitter had 2 to 3 data centers, and was moving some stuff to GCP. Not sure if current state of things.

> and was moving some stuff to GCP

my condolences

>Mr. Musk is also considering shuttering one of Twitter’s three main U.S. data centers, a location known as SMF1 in Sacramento, which is used to store information needed to run the social media site, four people with knowledge of the effort said. If the data center in Sacramento is taken offline, it will leave the company with data centers in Atlanta and Portland, Ore., with potentially less backup computing capacity in case something fails.

https://www.nytimes.com/2022/11/18/technology/elon-musk-twit...

There are currently three data centers, hosting all of the real-time and most of the batch production load. Ad-hoc load and some production batch load was migrating to GCP, but this was being significantly curtailed in the months before the acquisition closed because it turned out to be very expensive, much more so than anticipated and more than the equivalent workloads had cost in the data centers.

Thanks! Yeah the other guy is right

Kudos for nice work!

What did you make of Mudge's report regarding resiliency of data-centers?

> Insufficient data center redundancy, 59 without a plan to cold-boot or recover from even minor overlapping data center failure, raising the risk of a brief outage to that of a catastrophic and existential risk for Twitter's survival.

- https://techpolicy.press/wp-content/uploads/2022/08/whistleb...

Thanks! There was a lot in that and I didn't follow it closely. Honestly that always confused me about the dc failing prep. Like I get that it would be a good idea to have a documented plan but at the same time, we did lose a whole DC and figured out how to recover(not that it was that easy) Im not sure having a cold boot prep plan would have helped that much. The things that failed that I dealt with personally during that event, I don't know if I would have foreseen. The site went on afterwards so it didn't seem to be existential.

So you still work there? Or did you quit recently?

I left over the summer

Weren’t Twitter moving out of Mesos to K8s?

Yeah there was a few teams trying to make it happen

Reply on news.ycombinator.com