Hacker News Reader

Comment by emodendroket | Hacker News Reader

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people became really invested in the idea that a site cannot possibly stay up without dedicated SREs, as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations.

View on HN

I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now:

1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that have 0 people left.

2. More than the technical issues, betting that Twitter will go bankrupt is the easiest bet one can make. Musk saddled Twitter with a shit ton of debt - even if things worked as they did before he had to cut tons of people due to the debt burden.

The issue I see is that #2 directly works against #1. Musk has said it will be lots of intense work adding new features to try to raise revenue. But making a ton of changes, probably with lots of shortcuts to get them out the door quickly, especially when so much institutional knowledge has walked out the door, will make keeping the site stable even that much harder.

Wait Musk saddled Twitter with debt? Please elaborate...

It was a leveraged buyout. Musk borrowed $13 billion to complete the deal, which Twitter needs to pay back.

And if he doesn’t, it might effect tesla very negatively (he used his tesla shares as collateral)

> he used his tesla shares as collateral

No he did not. It was initially proposed but there's no margin loan against his TSLA holdings in the final deal. For the debt that Twitter took on in the buyout the collateral is the company itself. There's zero connection to Tesla in the actual financing.

https://www.cnbctv18.com/business/who-is-funding-elon-musk-i...

It was a leveraged buyout. They are a deal where the company borrows money and combines it with the investor's money to buy all of the shares back.

https://www.washingtonpost.com/business/elon-musks-twitter-d...

Yeah, it's private equity 101. I can't believe it happens. A bunch of MBA sharks get loans to buy something with some chop shop plan or other ruthless scheme to carve up or spitshine a company.

The loans are then assigned to the company they bought, rather than the core sharks ... uh investors. The "investors" then repackage/selloff/spit shine as necessary to get it resold to some sucker. The investors get THEIR money back with a profit, leave the debt to the company and the sucker who buys it, and look for the next victim/target.

Musk likely went the rapacious route because he made a dumb move, and this is the only way he's getting the money he committed to the deal back.

Oh yeah, and when they buy the company, they are in charge of it. So they can pay themselves whatever they want in executive bonuses / etc.

You make it sound like it's a bad thing.

The people who buy the debt know what they are getting into. (And if those 'suckers' don't know, honestly, they shouldn't be investing in junk bonds etc.)

It is not a bad thing as such. The main problem comes when a business is stripped of all its assets and/or weighed down with more debt than it can repay and fails as a consequence, leaving employees out of a job and communities without a formerly productive enterprise.

Well, business people take risks, and sometimes those risks turn out badly.

It is more that it tends to beat raw deal for the purple who lose their jobs for no good reason. The Twitter people will likely mostly land on their feet but most leveraged buyouts aren't affecting highly compensated employees with highly in-demand skills.

do they though?

what if the debt is sliced up and repackaged and rated AAA fraudulently?

The problem there is with whatever rules that make AAA ratings magic.

Why is that possible or legal? It feels like a loan for purchase of a company necessarily should belong to the purchasers of the company and be paid by the sale of the equity they bought or profit. This way just feels… rigged.

You might enjoy this book if you haven't read it already

https://en.wikipedia.org/wiki/Barbarians_at_the_Gate

They are paying for the privilege of the direction of their wise new overlords, I suppose.

> This way just feels… rigged.

Well, yeah. Like the rest of the financial system.

because libertarians took over the government in about 1980 , thats when this stuff really got going.

> Yeah, it's private equity 101. I can't believe it happens.

Why not? It's pretty good for all the stakeholders that have a say.

The old shareholders sold for a nice price.

The new ownership didn't have to pay for the whole thing.

The lender gets to charge a pretty good interest rate because there's a good chance of default. If they're lucky, they get repaid; if not, maybe they made enough in interest to make it worthwhile; maybe when it defaults, they'll be able to make something worthwhile out of the wreckage they got at a nicer price.

Leveraged buy outs aren't great for stakeholders that don't have a say. Employees usually get new terms worse than the old ones; in this case, there's been some severance at least. Customers get a promise of a big bang bankruptcy in the near to medium term, rather than a slow fizzle. Sometimes companies with a large payment they can't make can restructure, and sometimes they shutdown with little notice. As a private company without public accounting reports, there will be a lot of guessing about revenue and debt service.

Which goes to demonstrate the very sorry state of our society, a society where employees are not (anymore) part of the "stakeholders that have a say" group.

Because at the end of the day, as you mention, it's the employees and their families that will suffer the most. But as long as those employees don't have board seats while strikes and labor-related physical protest movements have become a thing of the past then I guess this is the reality we'll have to live on for the foreseeable future.

It’s quite bad for the business as a going concern and its ability to serve customers and employ employees when it suddenly, and for no obvious reason, takes on tons of debt. Money isn’t free and loans demand payment.

In Twitter’s case, its $13B in new debt on the balance sheet means that, every year, they have to come up with $13B times the interest rate in additional revenue and/or reduced cost merely to be in the same place profit/loss-wise. Elon already massively overpaid what Twitter’s business-as-usual can generate even before accounting for that $13B; as a result, the post-acquisition Twitter has to try a bunch of negative-expected-value moonshots in the unlikely hope that one of them hits against the odds and the others don’t turn out fatal, because doing nothing or iterating sustainably kills the company via debt service.

In a lot of cases the best solution would be for the company to declare bankrupcy, reorganize, and discharge the debt by convincing creditors to take pennies on the dollar and a share of the resulting smarter-run healthier company, and a judge that the plan is reasonable. However, Elon both poisoned that well by firing people, angering advertisers, and bumbling around product-wise, and also staked a bunch of $TSLA that would need to be liquidated to go through with the bankruptcy.

Yeah, I find that easy to envision myself, but it's not the kind of spectacular, instant failure that's been predicted at all.

Generally agreed. However keep in mind that bankruptcy is a purely financial event and doesn't necessarily have to have any impact on operations.

Airlines are notoriously for going bankrupt regularly.

Depends.

"Chapter 7 of Title 11 of the United States Code (Bankruptcy Code) governs the process of liquidation under the bankruptcy laws of the United States, in contrast to Chapters 11 and 13, which govern the process of reorganization of a debtor. Chapter 7 is the most common form of bankruptcy in the United States."

https://en.wikipedia.org/wiki/Chapter_7,_Title_11,_United_St...

Airlines get free government money though. Will twitter? If not, Twitter will need to borrow more money and at higher interest rates compared to market rates if they go through with bankruptcy.

Funny thing about the bankruptcy hypothesis is that the owner of Twitter is the richest man in the world and could literally pay old twitters expenses for decades even if it never made a penny. If we are just talking about the debt payments, he has a 200 year runway.

Pure copium.

Actually be probably can't. Musk's wealth is largely on paper, as Tesla stock. Which is enormously overvalued and has been plummeting since he bought Twitter. If Twitter needs cash injections, he has to sell more Tesla, which will depress the price further.

200 years is long enough to sell it off without depressing the price.

A billion dollars a year of stock sales is a huge amount of guaranteed supply to the market. And it's non-optional: it has to be a billion dollars a year. But Tesla isn't worth anywhere near it's current market capitalization either.

For context, Elon sold 36 billion worth of Tesla this last year alone.

If you don't think he can keep selling 1B a year, I don't think anything I say will convince you.

The idea that the banks will somehow repossess Twitter from an unwilling Elon is detached from reality

I think that share quantity and value are very separated concepts. At some point Elon loses control of Tesla and that's it: Tesla's price has been tanking, and it's still on its way down to a sane valuation.

> But Tesla isn't worth anywhere near it's current market capitalization either.

In what sense?

The Top 10 manufacturers for 2021 were[1]:

1 Toyota Motor Corp 10,495,548 (same) +11.8%

2 Volkswagen Group 8,610,100 (same) -5.5%

3 Renault Nissan Mitsubishi Alliance 7,680,014 (same) -1.3%

4 Hyundai Motor Group 6,667,085 +1 +5.0%

5 Stellantis 6,583,269 +1 +5.2%

6 General Motors 6,291,000 -2 -7.9%

7 Honda 4,121,000 (same) -6.5%

8 Ford Motor Company 3,942,000 (same) -5.9%

9 Suzuki 2,763,000 +1 12.9%

10 BMW 2,521,514 +1 +8.5%

Notice who's not in that list?

Now let's look at market capitalization[2]:

1 TeslaTSLA $568.87 B

2 ToyotaTM $202.86 B

3 PorscheP911.DE $101.44 B

4 BYD002594.SZ $86.39 B

5 VolkswagenVOW3.DE $84.30 B

6 Mercedes-BenzMBG.DE $69.28 B

7 General MotorsGM $57.48 B

8 BMWBMW.DE $57.39 B

9 FordF $56.60 B

10 StellantisSTLA $48.87 B

There is no possible way Tesla is worth more then double what Toyota is, while shipping about 1/10th of the vehicles. The Price-Earnings ratio is 56, vs Toyota's 9.

By every possible metric, it is vastly overvalued compared to it's profitability, and there's no reason on the horizon to think it will suddenly catch up since all those automakers are moving directly into it's space.

[1] https://www.carexpert.com.au/car-news/who-won-the-automotive...

[2] https://companiesmarketcap.com/automakers/largest-automakers...

It's borderline a meme stock at this point. Tesla is valued as if it will forever be able to sell every car they make, on earnings calls Elon blows off any questions about demand. In the last few months though used car prices for Tesla have plummeted, I think they're finally about to run into demand issues as they get more supply online

If you want to compare companies by 'how big' they should be, you need to compare enterprise value, not market capitalisation.

If you think Toyota's stock (or bonds etc) will increase in market price relative to Tesla, for a fee financial markets will let you bet on that conviction and make money, if you are right.

I also want to add a #3: the crew that's left is probably on-call 24/7. My thoughts are with the poor souls on that rotation (if your team even still has a "rotation")

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

Definitely stealing this as the right way to frame the issue. What would normally be a small kitchen mistake turns into no longer having an apartment complex.

He also sold all the smoke detectors.

Credit where it’s due though: he did supply a spare sink.

You should generally ignore sunk costs.

My analogy.

Someone purchased some land for $1. Built a house for say $100. And now spends $100,000 a year making it the perfect place to rent, receiving $100,000 a year in rent.

Someone comes along and borrows $1m to buy that house. They feel ripped off but eventually are force to go ahead with the purchase. As a result they have to pay $100,000 a year in interest. They need this thing to be profitable!

To do this they need to cut back on the $100,000 a year spent. They decide go go in quickly and so email all the services saying "go hard or go home". So the plumbers, tradie, cleaners etc that don't like it leave.

As a side hustle also charge visitors to the house $9 to be allowed to wear their bowtie they used to wear for free.

Some of the people do maintenance jobs and improvements. They keep the termites out, fix subsidence issues, and so on.

And the house didn't fall down within 3 weeks of it being purchased.

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to.

So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building. But eventually people are going to start turning on stoves again… and then…

This is correct, but also: a sudden decline in maintenance is a kind of change in its own right. Even automated processes have humans in the loop and manual sign-offs; there's always some cronjob or short-lived certificate somewhere that a human was dutifully maintaining.

Those things aren't going to fail any sooner than they would have anyways, but they're going to fail a lot harder due to the loss of institutional knowledge.

Welp, their TLS cert expires second week of the new year. I really hope for them that’s an automated process.

Except that he at the same time demanded that people invent an entirely new dish by the end of the week, and now they are scrambling to try to figure it out. Already the DMCA auto-takedown bot is apparently broken and people are posting entire movies on Twitter. I would expect other peripheral systems to start breaking down as nobody is maintaining them even as other parts of the system are being changed.

oh noes! Not the DMCA auto-takedown bot!

Said no-one in the entire world except a hand-full of Hollywood studio owners.

That bot shouldn't have existed in the first place, but I know that that falls under "just world fallacy" and is a naive thought.

I interpreted the GP's comment less as a moral claim ("the DMCA bot is good") and more as a claim that the DMCA bot's failure is a strong indicator of internal instability (given that it sits directly at the intersection between Twitter's profit interests and microservices architecture).

Put another way: being unable keep a little bot running, one that keeps an entire industry happy, doesn't bode well for other components of the service.

No it’s proof that people will take anything and run with it. This bot likely had low priority and that’s all

Run with what?

It seems self-evident that the bot was considered low priority, since it isn’t working anymore. But nobody is disputing that: they’re saying that the fact that it is low priority does not bode well.

> This bot likely had low priority and that’s all

If it was a prerequisite to land $100M ARR from all the media properties’ marketing budgets to advertise the multi-billion dollar pipelines of the movie and entertainment industry, that lil’ bot was the gate to $11,415 per hour of revenue at risk if its uptime failed to sufficiently please the attorneys and auditors from those customers.

Well it is exposing them to significant legal risk if they no longer comply, isn't it?

I mean, does Twitter want to be a party to a copyright lawsuit? If not, following legitimate looking DMCA notifications (and legitimate looking DMCA counter-notifications) and responding to suponeas as necessary gets you an affirmative defense for copyright infringement.

You may not like it, but having a bot do that probably saves a lot of legal hassle.

Content providers cannot be held liable for user generated content under section 230. Try again.

§230(e)(2) says

> (2) No effect on intellectual property law

> Nothing in this section shall be construed to limit or expand any law pertaining to intellectual property.

If section 230 from the CDA of 1996 provided immunity from copyright claims, there would be no reason to include procedural requirements for processing claims in the DMCA of 1998.

Hm? Intellectual property is explicitly carved out of 230, and even if it wasn’t: it isn’t user generated. Content providers are regularly found liable for infringement on their platforms, especially when the plaintiff can demonstrate willful negligence (which in this case would include discontinuing a seemingly effective scanning system.)

Twitter is a global company.

Many jurisdictions take an even harsher line when it comes to being complicit in intellectual property abuse. We saw this famously with The Pirate Bay, Napster etc.

youtube literally been sued over this exact issue multiple times.

I wonder whether this handful of Hollywood studio owners will have any influence on Twitter's revenue at all...

Well, Disney won’t care why their copyrighted material is publicly available, noone likes this sort of copyright, but if Elon wants to avoid huge fines he better (make someone) fix it ASAP.

2FA as well

Are we in support of DMCA now?

I don't think you really think that's what OP is saying.

> Twitter isn’t shipping many new features right now

Is that true? I thought one of Elon’s big pushes was launching the whole Blue Tick subscription thing. That doesn’t feel like a small feature.

Given that Twitter already offered premium API access, they've got billing in place, so now they add a new form that, once your credit card is verified, flicks a boolean on an account that was previously flicked by another process.

It might not be small, but it's not exactly huge.

Blue is paid by in app purchase that seems to haven't implemented until that. Its functionality is far from premium API access.

As I said, a new form.

They already had billing integration, even if you're accessing it via a new route, and they already had a boolean on your account.

Like I said, not small, but not exactly huge either.

Except there was a crapload of social engineering testing around this feature that was just skipped altogether.

Suddenly that simple change had the potential for catastrophic consequences.

Yep, doing it right would've been harder, but Musk was after doing it right now.

I guess you could say doing it right wasn't exactly rocket science.

/golf clap.

It’s more than that. For one it’s not an existing boolean, there are now two different kinds of blue tick that are presumably stored separately. Blue is also supposed to give the user fewer ads (while making them more relevant) as well as additional weight in feed ranking algorithms. It’s also intended to be offered worldwide which adds a lot of complication to things like payment flows.

I’m not saying it’s going to bring the site down tomorrow but that one feature touches on a lot of services.

I imagine their premium API was also available worldwide.

Okay, so two booleans, and checks of those booleans in a couple of algorithms.

This still isn't a huge change, it's not completely new functionality. Once again, not saying it's small, but it definitely isn't huge.

Clearly worldwide payments is an issue otherwise they’d have rolled it out worldwide day one, and they didn’t. There must be something holding that back.

Plus I really don’t think you can compare B2B payments for premium API access to end user payments. Not least because they aren’t going to be going the same route: a huge number of them will be via Apple or Google in-app purchasing. Ask anyone who works with those systems, it isn’t a quick plug and play job.

In general though, a new subscription tier, feed algorithm changes, UI changes… if these aren’t, what is a big change in your book?

> Twitter isn’t shipping many new features right now

When was the last major new feature? The site has always seemed pretty stagnant.

They outright built me-too versions of both Clubhouse and IG Stories, and a lot of stuff trying to encourage people to be nice.

I was just about to make the same analogy. The chance that anything will break when there are no new commits is way lower

He rolled in, sold most of the fire extinguishers, and made a big show of trying to make cherries jubilee while shoving his dick in one of the few remaining fire extinguishers. Let's be clear, it isn't just the erratic layoffs it's Musk's incessant meddling that's going to be Twitter's downfall. He literally took down SMS based 2FA because "microservices bad". He fired the payroll and tax departments (HR too?). He's scared off Twitter's main source of income while saddling it with significant debt.

As an SRE I would have been shocked if Twitter failed catastrophically (well moreso than broadly disabling authentication) in short order. However failure is pretty much inevitable at this point given the damage that E-Lon is actively doing.

Whatever. Twitter and Musk deserve each other.

> However failure is pretty much inevitable at this point

I'm pretty sure there will be no failure at all, and Twitter will work just fine.

There have already been failures. If I recall the system for sending two factor auth codes via SMS was down for the best part of a day.

Not the end of the site by any means but cracks are showing.

Well, define 'failure'. Minor outages like the one you are talking about were happening from time to time long before Musk bought Twitter, and it even suffered long outages frequently - remember all those fail whales?

I meant that there will be no catastrophic failure that will permanently (or even for a few days) stop Twitter from working at all.

failure. noun. with no security team, hackers are able to get in easily.

everyone's DMs leak, all the anonymous accounts have their identity revealed, and all Twitter's clients (advertisers) have their bank account info made public.

Yeah, I don't think anything remotely close to it will happen, unless some of the fired developers have left themselves some backdoors which they'll give (sell) access to it to someone.

does this stuff happen automatically? is there a robot that goes out and reads about all the new zero-day exploits and patches all the software without human intervention?

On Hacker News, everyone's a comedian! And, yeah, as the sibling comment pointed out there've already been failures as a result of some musky action. While Twitter isn't likely to fail on its own, E-Lon is actively causing problems. You need people to deal with that, and even if he had motivated, relevant, and competent engineers… how long will they stay motivated without a paycheck?

Let's not forget that whatever code monkeys are left are now personally liable for running afoul of the FTC. Whatever motivation they may have now will run out pretty damn quick once they stop getting paid.

Pretty sure that the developers that are left will not be liable for anything unless they are knowingly participating in criminal activity such as criminal negligence that is the direct cause of someone getting seriously injured or killed.

Generally speaking prosecutors want to target the highest level individuals responsible for directing such activity in the first place, not low level implementers who have little say one way or the other.

For now I'm talking about the FTC consent decree, so administrative penalties not criminal charges. Musky fired the folks who were responsible for ensuring compliance.

https://techcrunch.com/2022/11/11/musks-lawyer-tells-twitter...

I think that he will soon bring in new developers/support engineers who wouldn't have questionable loyalties and grudges against the new management.

How do you propose Musky does that with no payroll department? E-Lon walked back the mandatory return-to-office policy last week. Surely if he could find (or thought he could find) suitable replacements he would be pretty comfortable demanding RTO.

I'm not really familiar with how they do payrolls in USA, but I'm pretty sure it isn't some rocket science (pun intended) and can be done rather cheaply by an outsourced firm.

Also, why are you (and many others here) refer to Musk as "E-Lon"? Is it supposed to be a derogatory nickname?

It's simply an abbreviated version of Elongated Muskrat as far as I'm concerned. Payroll is easily one of (if not the) most complex systems at any company. It's not just statutory stuff but personnel stuff as well. With Twitter you're not just dealing with 50 states and the feds, but with every other company in which Twitter has (had?) employees. There's a cottage industry of payroll firms precisely because payroll is so obscenely complex.

Even if you outsource it you'll still need people within your company to manage your service provider. At one company I worked for they got all of their outsourced HR+payroll for free (indefinitely) because the provider (Gevity) consistently fucked up everything they touched. This was at a company of like thirty people.

If you're suggesting Twitter can simply outsource payroll, sure. But you do that before you fire your whole payroll department. You still need people to handle the transition.

Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.

Whether that will spell disaster when there's a fire depends on whether the building had too many fire extinguishers to begin with and whether the boss can buy new, better fire extinguishers to replace some of them before there's a fire.

> Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.

If we're deep-diving it'd be closer to say that he rolled in and sold 80% of the stuff, largely sight-unseen, and if a fire breaks out he'll find out how much of that stuff was fire extinguishers.

Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own.

And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."

I don't even doubt that they did a lot of work, but as this article suggests, that's part of the reason why we _wouldn't_ expect the site to just go down in a couple days. If all the servers needed constant manual massaging that would speak more poorly to their work than the other way around.

I guess that highlights that for some time we just won't know the actual effects and everything right now is just posturing and speculating.

If in six months (or suitable time period) we see stagnation in releases and technical issues increasing we'll have some data to infer from.

I love this comment. It's so salient. It's probably hard for a lot of people on this board to digest but, I feel like most know where you're coming from.

It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees.

It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail.

Why? What's driving that response?

The only way a herd can move is by becoming confident that it is moving.

> put their reputation on the line saying Twitter wouldn't survive the weekend.

Was that a real thing those people were saying?

Genuinely asking, as I don't follow any of the former Twitter engineers on, well, Twitter, and if there were any such posts/articles on here I must have missed them.

There are a lot of places where the systems would start to fall over if 70-80% of the team departed. Especially since a lot of folks left on bad terms and/or were suddenly terminated. It was the opposite of a smooth handover.

So it wasn't unreasonable to think that Twitter would begin experiencing problems.

I didn't think that Twitter was going to literally have an unrecoverable system crash and permanently shut its doors, but I thought we'd see some outages or partial breakage over time, which is basically what has happened albeit in an admirably mild way.

    It comes off as a kind of desperation, as 
    though they need Elon to fail.

You don't need to pathologize it, like it's some... deep weirdo psychological yearning. Some people think he's a jerk and wouldn't mind seeing him fall on his face!

If a person was staking some significant part of their emotions on their feelings for Musk, yeah, that'd be unhealthy.

But I think you are significantly overestimating the emotional weight behind 99.99% of the half-baked Twitter quick takes. It's okay to not like a guy!

Just not like him yes, but also probably feel some desire to defend their profession.

Sure, everybody wants to defend their position.

But the parent poster characterizes those disliking Elon Musk as suffering from "psychosis" and "desperation."

It's a an unfortunately common, passive-aggressive ad hominem tactic you see on the internet and elsewhere. "People don't like thing or person XYZ? Oh, they must be mentally ill and losers who spend their whole lives obsessing over that thing/person"

HN should be better than that.

> Why? What's driving that response?

Experience.

Many of us have worked at companies where there is a lot of duct tape holding things together and when you let go of entire teams (not just a large percentage) then it isn't unreasonable to be pessimistic. Especially when you know that in order to fix problem A you need to take B, D, E, C corrective actions in that order. And you learnt that through years of things going wrong.

More so at companies like Twitter where they never really reach a steady state. You constantly have large fluctuations in system stress e.g. World Cup, Trump rejoining etc.

Just curious, among the critical stuff just holding on, was there also a whole pile of departments and teams doing work that could cease tomorrow and the company would blink and move on?

My experience is both exist at the same time because the leadership teams don't actually know what core business is or are busy building empires and resumes.

    Just curious, among the critical stuff just 
    holding on, was there also a whole pile of 
    departments and teams doing work that could 
    cease tomorrow and the company would blink 
    and move on?

I get what you're saying. For any given team with a public-facing product you generally have perhaps 20% of the staff keeping things running and the other 80% of the staff is working on new features, reports, enhancements, customer support, whatever. You could eliminate them and while it would diminish the company in various ways, the other 20% could certainly keep the lights on.

However it's worth noting that's not what happened at Twitter; there were very specific and explicit reports that the "keep things running" teams were hit just as hard by layoffs/resignations as the other departments.

So there was real justifiable concern there.

There's also a lot of things that can go wrong during sloppy and abrupt handovers. Like... you fired the guy who manages the domain renewals. In the chaos of transition nobody picks this role up. One day 18 months later you realize "twitter.com" has expired. Or whatever. Even if the remaining staff is sufficient to keep things running, there are thousands of these little process interruptions.

What is with all the people this year who insist that because something doesn't happen right away, it can never happen?

The only prediction anyone made was that the World Cup was historically a period of very high load, so if something was going to go wrong soon, the weekend would be the first vulnerable time, and the World Cup finals will be the next.

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivations rather than deep knowledge of system failures.

I had MySQL running on some bare metal for many years without a restart.

I was terrified to update the kernel at that point, knowing that system disk had been running continuously for many years, and had no faith it would restart successfully.

Finally got two new servers to replace these (with these new SSD things!) and after migration, sure enough, one of the old servers failed to boot.

Having single point of failure, and also not even knowing if it will even come up after reboot is a horrible way to run the service

Even if your mysql instance and hardware had run indefinitely, if a table is being written to it will eventually run out of disk space or key space and crash. How long it will take depends on the application but it will happen eventually and if no one is around to fix it...

I have a piece of Rust software that has not gone down in its entire lifetime.

Me too. It’s a hello world web service I wrote while reading a tutorial after work this afternoon.

Thanks for the laugh I needed that

By definition, no software goes down until its lifetime ends.

I would say the lifetime of a piece of software is different from the lifetime of the process running it.

Good for you!

Now, if your Rust code was a distributed system that handles spiky loads from ~330m users, and processes petabytes of data, then I'd consider your comparison relevant to Twitter.

But I'm going to assume it's not relevant.

P.S., I've written Java services that never went down, because they had a well defined domain and all potential errors were handled. But, I'm not about to compare that to all of frigging Twitter.

I wasn’t attempting to compare anything to Twitter…

The infra usually matters way more than the code. RAM or a disk will typically fail before the Linux kernel, and it's written in the boogeyman language.

> all software goes down and requires specialist intervention eventually

Well, that’s it, isn’t it? How many software systems need to keep running for Twitter to remain more or less functional?

If there are 10 critical systems that are running at four 9’s, you’d expect 3.6 hours of downtime a year, or about 90 days of uptime at a stretch if I have my math right.

If there are 100 critical systems running at 3 9’s, you’d expect 2.5 hours of downtime per day.

So yeah, all software should keep running. But it doesn’t. And something like Twitter isn’t “a software”, it’s a very large assembly of lots of different software systems and the exponential math that dependencies create.

Some of those 9s come with an assumption that someone is around who knows what buttons to push when it really goes awry.

Yep, and when one of the SEVs rolls around that would have been small (say 5m of downtime fixed with a flag flip), it instead will have a nontrivial chance of escalating into a major multi-hour/multi-day outage without the right institutional knowledge.

I'd guesstimate that Twitter probably has dozens of services that are in the critical path of an average user interaction. It's hard to keep even logically optional dependencies truly optional in large scale systems involving many people.

However Twitter didn't die in the past when fail whales ruled its day, so they probably won't kill it now. It's just not that kind of business. (In contrast, a one hour outage had me directly apologizing to our largest customers on the phone). That said, Twitter can only be unstable and lack feature growth for so long before something else takes its place, so Musk is on a clock.

It has effects on engagement, retention, and ad revenue though.

Right, but Twitter wasn't a healthy business (in the sense of being profitable most years) in the first place so it's not beyond the realm of possibility they took reliability further than made sense. Anyway they now have a huge debt load that changes the calculus regardless.

> Right, but Twitter wasn't a healthy business

I don't know why this nonsense keeps being spread.

a) Steadily increasing revenue, DAUs etc and benefited immensely from COVID.

b) Were on track to being sustainably profitable.

c) Starting to get traction in stealing ad spend from Facebook who had their own issues after Apple's effects on attribution reporting.

The company was in a much better shape than many other similar SV companies around.

Reminds me of an Alan J. Perlis quote:

> Is it possible that software is not like anything else, that it is meant to be discarded: that the whole point is to see it as a soap bubble?

http://www.cs.yale.edu/homes/perlis-alan/quotes.html

> I think a lot of people just jumped to the second level based on political motivations rather than deep knowledge of system failures.

Anyone who has ever been oncall can intuit how often stuff breaks in big or little ways. Sometimes it's transient and goes away, sometimes it can be filed away to be fixed in the next year, but sometimes, it turns out to be an all-hands-on-deck crisis for a team, or 5.

> The default, naive assumption should always have been programs keep running indefinitey on their own.

...for people who understand software to some extent. I get the feeling a lot of people see it more like a hamster wheel, where once the developers are gone it immediately starts noticeably slowing down as it stops (and are confused when that doesn't happen).

It wasn't only news outlets. A majority of my... politically-noisy tech friends on Facebook went through a recent phase of intensely posting about Twitter on fire and collapsing.

Now just because they're "in tech" doesn't mean they have any idea about Twitter, but they should at least know enough to know they don't know what's going to happen, but obviously they're not actually using their brains when posting comments like that. Point is a lot of people opposed to Musk have been participating in a spiraling echo chamber of fairy tales and wishful thinking, it's not just journalists (although clearly they're printing lies with ulterior motvies too, as usual).

I'll put my cards on the table and say I would have been just as happy to see him fail as anyone else, but it did begin to take on the tenor of the constant, never-came-true stories about how any minute now they were going to spring the trap on Donald Trump and he'd get thrown out of office.

e: I did not mean for this to be an invitation for everyone to argue about the merits or demerits of Donald Trump.

I personally have somewhat mixed feelings. I do enjoy a good dumpster fire like any good internet denizen ( and I won't deny that there is an odd level of.. excitement maybe.. that one major social platform may go down ). I have no love for social media as some of my posts may indicate and for a variety of reasons including the amount of oxygen this creation takes and the amount of confusion it causes in general population, it would not be the worst thing ever.

That said, the sheer amount of weird emotion attached to this particular billionaire is just perplexing. Only few years ago he could do no wrong. Today, it seems, he is being ganged up on including by some of his biggest internet fans. And media oblige for whatever reason. I dislike billionaires, but I dislike them as a caste in our societies. I don't see Elon as a savior of mankind or the devil, but those emotions that swirl around him are made to make him look like one or the other.

Somehow, one is supposed to pick sides.

Trump seems to create the same emotions. It is truly fascinating.

I was about to say that you're talking about two different problems. But, no, what's happening to Musk is the same thing that happened to Trump. Their inner circles knew how rotten they both were, but outside of that they were largely unknown.

New Yorkers despise trump and have for ages. From tenants, to contractors, to business partners they all know what a shitheel he is. Outside of NY though trump was largely seen as an affable caricature of a rich dude. Case in point: NYC as a whole isn't averse to republicans and yet they still rejected trump in 2016.

Elon's the same way. Sure he called that diver a pedophile. But he also had a Tesla employee hounded until he fled the country. Musk had the guy swatted for fucks sake (or tried to at least). Or what about the journalist who pissed him off and ended up having some Tesla features disabled on his car. Or his ex-wife who talked about being treated like an employee. These things were mostly inclined to stay in the background until Musk elbowed his way into the spotlight. Now the whole world can see that he went full on Howard Hughes.

<< Outside of NY though trump was largely seen as an affable caricature of a rich dude.

This is a very interesting statement to me. I am trying to recall conversations with my wife's side of the family, who supported Trump ( initially, mostly ) and their impression of him was almost exactly that ( plus, "he is a businessman so he knows the game" or something to that effect ). I am in in Chicagoland, but now I am curious if any serious study was done on whether that is how he was perceived as ( ideally before it became socially awkward to state your political affiliation openly ).

<< Their inner circles knew how rotten they both were, but outside of that they were largely unknown.

And that is the other interesting thing. Prior to whatever caused this change of hearts, I only saw impressions of Musk that were similar to impressions of Gates ( you know the ones - saving Africa or something; I know it reads awful, but I really don't remember what his PR presents him as ) - visionary making cameos in odd cultural vignettes ( I think he did one in 'Big Bang Theory' ). The articles describing his work style were very slowly bubbling up.

<< These things were mostly inclined to stay in the background until Musk elbowed his way into the spotlight.

And became too much of a power center himself.

I will need to sleep on it, but I think I agree with you.

It's because even though Elon Musk is no right-winger, he voted straight Democrat his entire life until very recently but then he voted for a Republican. The haters will brook no deviation from their accepted political narrative, and when Musk did it the hordes of people with pronouns in their bios turned on him.

For those people it has nothing to do with technology, which is why their takes were so hysterical and wrong. It's 100% about politics every day, 24/7.

Dems didn't want to throw him out. Pence was a competent politician. Trump wasn't. They wanted him embarrassed, not out. Much like the Brits had a plan to assassinate Hitler before D day. Choose not to. They were afraid he would be replaced by someone competent.

> as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations

Not everyone has the same knowledge and skills. Not everything is documented. Not everything in the documentation is current and correct. (especially after recent changes) It's not even that they won't stay up, but depending on who left the company, the oncall response may look radically different and have different time to resolution.

Yeah, but I think a judgment is being made that having the service be less reliable is tolerable.

With that amount of staff change, I’m not sure that service would inevitably come back up at all, in case of a serious enough problem.

Riiight. But what happens when there's nobody on that oncall rotation? I've worked at FAANGs, and I'm just thinking what would happen if there was a problem with an upstream team and nobody there. Maybe it's ours now. Maybe there's a principal engineer who worked on it back in the day that's still on the payroll? Maybe we're just going to try to bring it back up with no idea what is going on? What if you have a bug that silently loses data?

> but it was never going to happen overnight and I'm not sure why news outlets were saying that

I don't think it's that obvious, it's one thing to just leave everything running but Elon was also talking about changes he was making (Ex. turning off a bunch of "microservices" because they don't do anything). If you turn off the wrong thing and don't have anybody left who knows how to properly turn it back on again, then you're in a pretty bad situation. It doesn't seem like that happened, but I don't also think we have enough information to say how close it was to happening.

This whole thing has been quite telling for me. I don’t use Twitter and am not a Musk fan, but seeing someone try something to move away from ad driven seemed like a good possibility for me.

Then I saw how bitter and nasty a lot of the online communities I belonged to were. I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue. I was met with people attacking me as an idiot with clickbait articles of “proof” that ad models were irrelevant to twitters problems etc, as if I was doing anything other than quoting musks official reasons for the purchase. Suddenly Twitter was this great beacon of graceful discussion and Musk has ruined it.

People just love to be mad, no one had anything nice to say about Twitter and in a flick of a switch they’re holding completely opposite opinions. We’ve always been at war with Eurasia, it seems

>I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue

In the computer industry, there are some famous historical examples of companies that announced their new product before it was ready, people stopped buying the old one, and they went out of business before the new version was done.

Not quite the same business, but a bit reminiscent.

Reply on news.ycombinator.com