Hacker News Reader

Comment by rdiddly | Hacker News Reader

Why would Twitter be constantly in danger of crashing without human intervention? Isn't it just a bunch of software that just runs? If not, why was it not properly engineered, and who needs to be fired for that?

Musk is embarrassing as always but he's right about one thing - engineers are costly, and for years now, the problem with Twitter as a business has been the huge outlay for engineering talent. And for what? No effect I can discern; the platform is clearly not a force for good in the world, first of all, but even looking at it strictly as a product, the product in fact gets worse every year and they're constantly discontinuing functionalities. So the whole "what has your code accomplished" show, it's inept but I know where it's coming from.

View on HN

You need to:

* Apply security patches - there's thousands of dependencies.

* Manage hardware/cloud resources according to volume of data you need to handle.

* React to the changes in operating systems/browsers you run on.

* Fix bugs - they are there, because simply there's not enough time, money, and need to write "perfect" software. Competition will not do that and will beat you to the market.

There's no internet connected software, that can "just run", because world around is changing and you must catch up.

Fells a bit weird to explain that on HN :-|

This is all needed to continue to iterate on software. It’s not as needed for the ossified state Twitter has been in for many years, as the parent pointed out. Acknowledge there is _some_ work needed, maybe refreshing certs (though software systems I have built have always been set up to do that automatically), apply security patches, keep an eye on dashboards. But this is a job for a skeleton team, not 7500 engineers or whatever. It’s honestly kind of sad it took that many people to basically “keep Twitter up” all these years.

I believe that 7500 is total headcount, not just engineers, but yeah software development can be terribly inefficient, when all problems are solved by throwing money at them, like VC funded startups like to do.

We'll see what happens now, when there seems to be a drought.

There are also integrations with partners, new regulatory requirements, changes to support ownership's new priorities (and to undo the old priorities).

I don't work at Twitter but these things happen at every business.

What your saying is true for almost every web app. I don't see any reason to justify Twitter's complexity that requires 1000 engineers. Scale is such a weak argument since their entire product paralelizez very easily.

I don't think you understand the fanout/fanin problems their product has.

Twitter's raison d'être is allowing folks with millions of follows to see their posts in near real time, then monetize it with ads. That means analytics, billing, and more. From pretty basic requirements you can create a huge amount of work, especially when you're managing your own infrastructure.

I'm not saying that it's easy, just that most work could be done by much fewer people. I'm not talking out of my ass, I have been working on scalable systems for more than a decade.

I'm reminded of when, out of sheer curiosity, I once skimmed the proof for Fermat's last theorem by Andrew Willes. It was a monstrous work of which I could understand very little, but that clearly couldn't have been made by a team of a hundred mathematicians working together.

It's not that it isn't possible for a single person to do a monstrous amount of work, but that it takes an organization to do a breadth of things and maintain it over time.

I also don't disagree that some problems are self-inflicted. Large organizations breed large organization problems. But large organizations usually grow from seemingly simple requirements and compliance efforts.

My team certainly got a lot more done before we required tons of process, safety checks, data storage requirements, etc., where we now spend a non-trivial amount of HC as overhead and avoid building other things entirely.

> I'm not talking out of my ass, I have been working on scalable systems for more than a decade.

Ditto, friend. I currently work in an organization that builds a billion user product with far fewer than Twitter does. However, we also depend on a battalion of internal services, and other shared resources (HR, recruiting, PR, finance, ...). Twitter maintains their own datacenters, are among the few users of Aurora and Mesos at this point, and purpose-builds a lot of their infra for requirements we probably don't have a great grasp on.

Twitter, like many products, is like an iceberg: the majority of the complexity is underneath.

Well, I think we're pretty much on the same page regarding where the complexity comes from.

For Twitter though I'm thinking though that they should have put a lot more effort in simplifying the systems/processes/data they store. This might even mean pushing to remove some business requirements that a PM might have introduced just for a promotion to reduce software complexity.

If you don't reward people for refactoring and removing complexity, I don't think you can just blame it all on organic growth.

If you're gonna condescend like in that last line, your spelling ought to be impeccable.

Security patches... so 10 full-time engineers ought to take care of that?

"Thousands of dependencies" was a mistake. The ideal is minimal complexity.

Any browser or OS decides to commit a breaking change (rare), you bring in contractors.

Scaling up and down (and out and in) is handled for you automatically by cloud services now without anyone lifting a finger. "Fells" weird to be explaining that on HN.

Bottom line, you don't need thousands of engineers to keep Twitter running for "the next few days to week or two (at most)."

When has the world NOT been changing? The leadership of Twitter changed also.

twitter is mostly run in their own datacenters, and does not have autoscaling. so capacity overall is more or less fixed, though how much capacity is given to each service seems to be human adjusted.

the lack of autoscaling at twitter, imo, is worrisome as it sort of hints that the system might not be as mature in the self-healing department as compared to other large tech companies. time will tell.

you are right though, you don't need thousand of engineers to keep twitter running a few weeks. you also don't need a driver for a car driving a straight highway (until it bends). how many engineers are needed for maintenance? who knows. I don't think we have answers, we just have what elon decided.

disclaimer: facts based on twitter threads/online reading, not a twitter employee or anyone with first-hand inside knowledge.

By what I’ve heard from people who worked there they do have autoscaling but it’s services running on things like Mesos clusters instead of someone else’s data centers. That can lead to some real cost savings but requires people to run, and most of them just left the building.

As as PM my headspace is very much split on this.

On one hand I totally get your arguments, it's what I would expect my engineering team to say and they are often right.

On the other hand I am watching this space utterly intrigued if somehow he manages to pull it off.

If he does it might say more about the engineering team left than him though.

To be clear - I don't argue that it's impossible to run Twitter with smaller crew. Hard to say without a look at the internals, but if I had to bet - yeah, probably there are inefficiences. But OP was coming from "you can leave it and it will just run by itself" angle, which is very much different thing.

Technical and support teams are needed to keep a site like this running globally. Good examples of the work from an SRE: https://twitter.com/MosquitoCapital/status/15935411779656785...

Other things which aren't necessarily needed to technically keep the code running, but without which you won't have an operating service:

* Sales and account management teams to keep and grow advertiser revenue, and all the support services around them (marketing, integration, etc)

* Moderation teams to keep illegal or dangerous content off the platform, and implement your moderation policy which you need for many reasons including appeal to advertisers

* Abuse teams to respond to the latest DoS / hacking / impersonation / harrassment / etc.

* Legal and ops teams to understand and implement compliance with laws, regulatory filings, etc across 150+ countries, and in some countries like the US, individual state variations.

You tell me how many people you can do this with at the scale of Twitter. I think > 1000 people are a skeleton staff just to keep the lights on.

You are making the same mistake Elon is making. You don't know enough about the subject to even be aware of how much you don't know and what you are missing. You might be very competent in another subject, but that doesn't grant you the right to write in such a demeaning tone.

I don't need to be granted that right, I was born with it, just like you with your far more demeaning and ignorant comment about things (me) you know nothing about.

Sorry, you aren’t born with any rights. Those that you may appear to have were granted to you by a society. Never forget that

Your view on individual rights is far from universal. Never forget that.

"We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness."

https://www.archives.gov/founding-docs/declaration-transcrip...

Or, alternatively:

"All human beings are born free and equal in dignity and rights." (from Article 1)

https://www.un.org/en/about-us/universal-declaration-of-huma...

And if society collapses and whatever hellscape that replaces it withholds those “rights” from you?

No way. If I were a solitary hunter-gatherer I would still have the right to speak. In fact I seem to gain more rights the fewer other humans are around, so obviously those rights are not coming from other humans. That's how you know they're innate. But if I'm alone there's no one to hear my speech of course, which actually sounds pretty good right about now. I hate this "community," so why am I on here talking about Twitter, something I have never given a shit about? Well I got sucked in. I logged out of HN months ago as a "participant" for what I intended to be the last time, but then as a reader I ended up logging back in so I could use the "hide" feature (because I'm not interested in the vast majority of articles) and from there was gradually tempted into making a small comment here and there, and then today just being sick and tired of hearing about the delicate majesty of Twitter, well I guess I wanted to call bullshit. Twitter is shit and I DGAF about it or Elon. Anyway sorry I didn't read any of these replies, some of them actually look like good ones, and good-faith attempts to answer the question. All I did was reply to the shitty ones. I've asked YC to delete my account, couple times now, but they continue to not do it, so I take it that's not a thing that is done? So HN is just like any other social media site I guess - an addiction you have to break. And not enough hamsters running in the wheels that power it, to provide for an account deletion feature. Anyway I apologize for taking my hate of you guys, out on you guys. I know that sounds sarcastic as hell but I mean it literally. If I hate everybody, that's my problem to deal with. Which I will do by forgetting it all, probably. So have fun y'all and go outside.

I'm born with the right to do whatever I want that doesn't violate the laws of physics.

The rights I appear to lose were taken away from me by a society.

I actually don't ever forget that.

You have a right to speak. You do not have the right to be taken seriously when you’re speaking outside of your areas of expertise.

Not a problem. I don't need to be taken seriously in general or by people whom I likewise don't take seriously.

Twitter is a complex application which serves a ridiculous amount of traffic to people around the world. That kind of system is constantly changing in response to user activity - it’s done in the same way that a garden is done. Musk just laid off the landscapers so what’s about to happen isn’t immediate but is inevitable.

A little shocking to see a comment saying 'why does it take work to keep a website running' at the top of a comment section here.

I think people are just surprised that it takes so much work because we thought things would get easier as time went on. It's actually the opposite though as more technologies have been developed. There's way more things involved in the stack and all of those can interact and break in their own interesting ways.

There is also just frankly more traffic to serve because of mobile devices, so more things break at scale.

I’d also note that expectations have increased, too. Consider for example how much people expect to just work when they type text into a search box, when the corpus is gigantic, in many languages, and constantly updating.

> Why would Twitter be constantly in danger of crashing without human intervention?

Kindof true. Platforms need a ton of maintenance - but usually the fires needing to be put out are because of other human changes.

Stuff like disk space, certificate expiration etc etc will bring down a platform without maintenance though.

This may come across snarky, but it’s unintended. Instead I’m trying to set out a sort of mental framework for answering the question from (slightly less than) first principles.

That’s a bit like asking “why does any large internet service need infrastructure maintenance?”

If you can’t immediately see the answer in twitter’s case, then you either haven’t yet applied your knowledge of how large cloud services function and need maintenance to keep from cascading failures, or you don’t have that knowledge in which case the twitter case can’t be answered without explaining first the general case of how any large software system needs regular oversight. But then, I think that even if you can’t think of an answer for Twitter, you can probably figure out a little of the answer for the general question.

You can start with asking yourself another question: why would Twitter (or any very large software project) pay $Millions for maintenance that isn’t required? Is there collective delusion on the topic? Or has twitter hit upon the absolute perfectly engineered platform ever? If the answer to either of those is “no” then a twitter system failure is only a matter of time.

How much time is a matter of some debate. The first time something hits a quota or something of that sort that requires a human input to ensure automatic allocations don’t accidentally get insane could be a pebble. Or maybe the first pebble is a minor unpatched security flaw, but twitter is a very sweet high profile target… whatever the first pebble is, it makes a slightly worse issue or multiple additional pebbles much easier, and you enter cascading failure. They generally happen slowly relative to the proportion of a total collapse, and then very very quickly.

So another question to ask is how much of twitter’s infra teams were pure bloat, how much were excess capacity necessary to cover average amounts of employee vacation/sick/leave that goes on at any time, etc, and how elastic the remaining staff can be (putting in extra hours, working in areas that are secondary skills because no one else is available) and for how long they’re willing to do it before burnout or better job offers with less stress and overtime, whether staff losses have already hit a critical mass to make this inevitable…

But I think this gives a general sense of why it can’t tick over forever on its own, or at least some inroads into thinking about the answer.

>Isn't it just a bunch of software that just runs?

the fact aside that bug-free "just a bunch of software" of the size that serves half a billion people has yet to be invented. All software has the nasty habit of running on hardware and real infrastructure and that at the very least needs some serious maintenance.

If your standard for proper engineering is that it runs without human intervention at all I can't think of any complex system in the world that fits that description.

Twitter is hosted on aws they don’t maintain their own hardware. This is a cope because Elon is clearly right. An application where prod begins to get buggy just because it’s left alone is poorly engineered. There should be CI/CD and prod should be extremely stable with the exception of non-trivial bugs that should not just start popping up by the barrel because a couple of weeks without new pushes.

Twitter is not "hosted on AWS". They have started to use various cloud providers (AWS and GCP, at least at various times), in combination with their own hardware.

Even running on AWS is not completely isolating you from hardware related changes. For example AWS is retiring both old instance classes for databases and Aurora 1 soon. This is a multi week migration project with a hard deadline for me and it's not even a huge company/service. For Twitter I suspect that kind of deprecation would take month(s) to fully roll out.

That's surprising to me, given how much effort AWS put into emulating old instances types using newer instances to still be able to offer instances with the characteristics of old instance types to customers. [1]

Do you have additional information about that you can share (maybe an announcement regarding this made by AWS)?

[1]: https://perspectives.mvdirona.com/2021/11/xen-on-nitro-aws-n...

It happens quite often. I'm dealing with deprecation of db.r3 for example which will be enforced in a few months. (I easily find the public announcement - you get an alert if using it) The instance sizes don't cause that much problem as such, but you still need to spend time assessing the new performance, updating reservations, figuring out the transferability of the old ones, etc.

I work in an unrelated company. Even without new software pushes and even for stable code, we need maintenance all the time.

A few examples:

- security holes discovered in the OS requires an emergency update, which requires lots of testing because updating the OS has a bad tendency of breaking even stable software (I've encountered that issue a few weeks ago);

- security holes discovered in third-party libraries/frameworks/..., which requires updating, re-testing, re-releasing stuff – and fixing whatever breaks because of undocumented changes in said library/framework/...;

- security holes discovered in your own code, of course, which also requires updating, reviewing, re-testing, re-releasing;

- monitoring your stack for misbehaviour, which could indicate a software problem (bug? license expired? SSL key expired?), a hardware problem, a resource problem (disk full?), an attack, or in the case of Twitter, one of your clients (the companies that build ads) misusing your tools and attacking you by accident;

- ... and once the misbehavior is detected, actually investigating and fixing the issue.

Sure, you can hop along for a while without anybody to handle these cases. But how long? Keeping in mind that Twitter is a high-value target for state-sponsored attackers (among others) all over the world, so any weakness will be probed and exploited mercilessly.

I run an engineering department so I’m not speaking from ignorance here. 0 days and bugs making it into prod is a red herring that isn’t part of the discussion. We’re talking about a production platform outright failing because it’s a sinking ship being kept with duct tape.

If Twitter is failing that easily, these engineers deserve to be laid off

Well, I can only speak from experience. I've encountered all of the points above just during the last 3 weeks or so. I've seen critical infrastructures in former companies being taken down by an expired SSL key or a full hard drive or a power outage or a DDoS.

Does your engineering department have a solution to all these problems that does not require human beings? Or are we talking of different things?

I would be interested to hear which web apps with 100,000,000+ users would run fine for weeks/months without hiccups if 80% of the people running it left overnight.

> This is a cope because Elon is clearly right. An application where prod begins to get buggy just because it’s left alone is poorly engineered.

Twitter is a top-5 near realtime web property. It has an enormous attack surface across nearly the entire spectrum of cybersecurity threats. It's hosted at least in part on prem. Swathes of it are programmed in Scala, which is a minority language that is hard to hire for. It makes money through ads, which involves significant operational complexity at scale. Its recommendations and trending features are built on machine learning models which themselves require data engineering pipelines to deliver continuous updates. It's been in continuous development for 16 years, with an accumulation of accidental complexity (aka tech debt) in addition to the ever-expanding essential complexity inherent in its functionality.

Those of us who have taken an oncall shift in the world's most complex distributed systems environments are just shaking our heads at some of the drivel in these comments.

> Why would Twitter be constantly in danger of crashing without human intervention? Isn't it just a bunch of software that just runs? If not, why was it not properly engineered, and who needs to be fired for that?

I would encourage you to read about “site reliability engineers” and the work that they do, in case you haven’t heard of them.

I can write software that runs today. I don't know about tomorrow. Disks crash, other services crash, networks go down. No matter how competent your Byzantine Generals are, sometimes you have to recover data that was in-flight during a crash because disks and networks don't just crash, they slow down and they lose packets and sectors.

Add on security patches and changes in architecture (I'm looking at some Java 1.3 code that runs INSIDE an Oracle DB right now, ask me how I feel about not updating your architecture). You may say that architecture doesn't change that fast. If the business is successful, it will happen eventually.

At scale everything is more complex. The hardware that runs the software can fail. Ddos happen. Network issues. A building catches fire. Etc etc.

Sometimes you have automatic recovery. Sometimes you don't.

All sufficiently complex control systems run in failure mode (partial or full manual override) approaching 100% of the time.

For instance this is why there needs to be a multi tiered (culminating with HN front page exposure) process for fixing automated account cancellations and app store rejections.

Reply on news.ycombinator.com