Hacker News Reader

Comment by minimaxir | Hacker News Reader

The hard part for Twitter is figuring out who's left that can fix it.

I disagree, the day he took over twitter people were saying "it wont be up this weekend" "it will be down in a week" "it will be offline in 2 weeks"... yet the reality is, its up and running well.

The system is running as good as before. The only thing I've noticed is theres more ad's than usual.

----

Edit: Currently there isn't any outage of Twitter as far as anyone knows. It looks like 3rd party apps have been kicked to the curb.

A ~16hr regional outage in AU/NZ is not indicitive of anything. In 2021 Facebook had a global ~7hr outage.

Since Musk took over we haven't had all the downtime and disruption everyone kept claiming we would have. Big companies haven outages too, but so far the death of Twitter isn't in sight.

Also: I hate Musk since his comments on Ukraine and Taiwan. So don't take my comments as trying to stick up with Musk. I'm not, I hope he loses all his money and becomes broke.

Consider the quality of engineer of the departed team that allowed Twitter to run without incidents for so long. Also, change is the most frequent cause of incidents, as no one was pushing new changes there were fewer chances of an incident.

Surely you're writing this in jest?

https://amp.theguardian.com/technology/2022/jul/14/twitter-e...

Reading that article, I am actually impressed that after that 45-minute outage in 2022, the next major outage was all the way back in 2016. I would say that is actually quite an endorsement for the quality of engineers they had.

It's not like the employees are some gnomes running the system by pulling levers so it is not surprising that rhe system seems to be running ok.

The problem arises at the point when there are bugs or other incidents and there is nobody who can fix them.

Australia and NZ experienced a 16h outage last week.

Was there someone to fix it eventually? Sure. Is 16h of downtime acceptable for a $44b business? I'm going with no.

https://www.gizmodo.com.au/2023/01/twitter-down-australia/

They've likely learned a lot from that incident already.

No, Twitter isn't gonna "blow up" overnight. That's nonsense.

If they hadn't purged the entire staff they wouldn't need to relearn the same lessons

> No, Twitter isn't gonna "blow up" overnight. That's nonsense.

It's also not something any of the parent comments claimed.

I take it no such outages occurred in the prior history of twitter?

16h is an awful lot for a social media site, even if region-specific.

Single digit number of hours happen from time to time, but double digits are not really a thing anymore. Maybe in early days of social media, not in the past decade.

Well, RIP twitter I guess. Although strangely if I search the news archives for outages with date set to before 2022..

You get what? Anything close to that long? Don't leave me hanging!

I did the same and found a Guardian article that claims no Twitter outage lasted even an hour since 2016, but I thought it was irrelevant because articles about outages aren't exactly top-notch in quality.

Did you find anything better? If so, do share! I'd be happy to be proven wrong!

> I did the same and found a Guardian article that claims no Twitter outage lasted even an hour since 2016

Actual Guardian article from 2016 discussing twitter outage lasting more than one hour:

https://www.theguardian.com/technology/2016/jan/19/twitter-d...

Amusing comments over there from readers.

You are right and I am wrong. And I'm sorry. RIP Twitter.

How is that against what I said? I said no outage lasted over an hour since 2016 and you disprove that by posting a 2016 article of an outage that lasted maybe two hours? (I don't know, the article doesn't specify.)

Is let's say five hours and sixteen hours within the same ballpark? I'm not interested in a couple of hours, my claim is not one lasted 10 or more in the past decade. If such a thing actually happened, I'm sure someone would've posted it by now.

> Is 16h of downtime acceptable for a $44b business?

After a 70% reduction in headcount, is a 16 hour downtime acceptable in a 5% market? I'm going with yes, executives from all major corporations are itching to press that button if only they could get away with it.

The problem with Twitter is not, was not, and will never be a technical one. It's a simple site with a simple tech stack, using standard technology to scale.

The problem with Twitter, and what Musk doesn't seem to understand, is a sociological and political one. Social networks are the crack cocaine of tabloid media, left to the whims of bad actors they can destroy democracies, incite lynchings, distribute hate and misinformation on scales without historical precedent. "Free speech" just doesn't cut it, social networks are publications not mediums, they chose what people get to see and react to, and the fact that the choice is made by an algorithm does not absolve them of responsibility for the consequences, especially if the algorithm is tuned for revenue maximization and easily gamed by bad actors to spread their content.

By firing the watchers, Musk is setting Twitter up as an menace to stable democracies around the world.

At the scale twitter exists right now it is anything but simple. Had it been built from scratch it could have run fine with its current staff.

But the current staff needs to maintain code from a MUCH larger staff. That's hard. Especially with microservices where every team did "their own thing" and some teams just vanished completely. You have "dark areas" where no one knows how stuff works and what's going on.

Normally when you downsize at this scale you do this in an orderly way. Every person hands off their work and responsibility. Spends a couple of weeks writing docs and instructions and then leaves. This was done quickly and violently. I'm sure there's many repercussions we don't see simply because the systems are simply running. Once the team looks at some containers and decides to pull the plug to "see what happens" we'll find out...

> “At the scale twitter exists right now it is anything but simple“

Elon should get real experts like Infosys or Wipro in to rebuild it.

Sometimes it's really hard to tell if people are joking or serious in HN...

They are seriously joking.

I thought adding /s would kill the joke ;-)

> Had it been built from scratch it could have run fine with its current staff.

So you are essentially saying Twitter has an over-complicated architecture with regards to its real technical needs, so Musk was right to axe the microservice zealots. Good we can agree on that.

The next step is to acknowledge that code and data-centers continue to work long after their original designers have left the company, that's what computers are, tools for automation. Keeping the lights on and services in a roughly functional state is an activity that requires much less people of relatively lower qualification, even if they couldn't have engineered it themselves.

Yes, there is a long term risk of code rot and entropy if Musk does not plan for a gradual transition to a better engineered architecture with competent maintainers. But we are not there yet, far from it.

Meanwhile, schadenfreude people have rooted for a twittocaplipse from day one, nay, predicted it as inevitable. There is absolutely no sign it's going to happen, and Twitter turned out sufficiently simple after all on the technical side.

People choose to use social media, as they choose all media. People are individually responsible for the physical actions they make, even if a little bird told them what to do. It may be easier to control people by limiting what thoughts they're exposed to, but a people isn't free if that's done.

It's all fun and games until you are slain in the street because people chose to regard you as subhuman after a little bird told them so.

Disseminating politically extreme misinformation kills people - look no further than Russia, where a state run propaganda campaign has made people no more stupid or immoral than you or me support a vicious war of aggression against their neighbors.

No, the killer kills physically. Being told to is no excuse.

No.

Facebook had downtime on all services, globally... for like 7 hours 2021, facebook is how many orders of magnitude bigger than Twitter?

And a regional outage vs the internet claiming twitter would be offline completely?

Half an order of magnitude bigger, at most.

A regional outage of that size is embarrassing whatever way you slice it.

Leaving all that aside, he’s driving a massive subset of his users away at a time when he really needs all the users he can get.

You can spin this however you like, this is not a good situation for him. He’s said as much outright and implied it repeatedly.

Since this time last year he’s said that both Twitter and SpaceX are likely months from bankruptcy, and Tesla has been flat out told FSD, the feature they’ve been selling all their cars based on (which he has said himself the company is worthless without), just isn’t happening.

This is dire for him.

> Since this time last year he’s said that both Twitter and SpaceX are likely months from bankruptcy, and Tesla has been flat out told FSD, the feature they’ve been selling all their cars based on (which he has said himself the company is worthless without), just isn’t happening.

As much as I hope Elon loses everthing. This isn't even related to downtime of Twitter. Volentarly shutting down twitter due to it being bankrupt is not the same as it falling over and having outages.

So I'm really not trying to spin anything. You're trying to take something unrelated and use it as justification as to why big bad billionaire do bad.

This is a fair comment, I'll clarify why I mentioned that.

Bringing up Tesla and SpaceX was only to point out that he doesn't really have any way to bail out bad Twitter decisions despite it being a (relatively) smaller company. There's nowhere to pull extra finances from to bail this out, and it's known he already went into debt to make the purchase.

I really don't want him to lose anything. I'd like him to go back in time to when he first installed Twitter on his phone and slap the thing out of his own hand and scream "NO! THAT'S A BAD ELON!"

In that case I agree.

I can assume one of the primary reasons for cutting staff and offices is to reduce costs and hopefully get cash flow positive asap even if it’s $1 positive. That puts him in a better position to stop incurring debt.

Maybe spray him with a water bottle to really discourage the behavior.

All the money in the world and what he really needed was a close family member to wave a rolled up newspaper at him disapprovingly.

Not to disagree with your general point, I just want to nitpick this:

Half an order of magnitude bigger, at most

I think you're overestimating Twitter and underestimating Facebook.

I don't know the best source for official stats, but a quick google suggests Twitter is ~300M users and Facebook is ~3B, so one order of magnitude.

And of course the new Twitter CEO used the interesting bargaining tactic of talking down his own purchase by implying many of those 300M users are bots... Joking aside, you could argue that very few of those 3B Facebook users are active, and you'd be right; but exactly the same can be said of Twitter and there's no good reason to think the ratio is any better or worse there.

> Since this time last year he’s said that both Twitter and SpaceX are likely months from bankruptcy

At least in the case of SpaceX the US government will more likely than not step in and nationalize them out of national security requirements.

It always good to have fallback solution, isn't it?

That's nice for the US space industry, but won't help Elon in the slightest with his money problems.

>SpaceX are likely months from bankruptcy

Companies months from bankruptcy don’t get to take a higher valuation in a down market: https://www.forbes.com/sites/qai/2023/01/03/elon-musks-space...

SpaceX is a great company, worth investing into, because of their ISP + phone telco business (lot of people are ready to pay a high price for that) and a very difficult space technology to reproduce, nobody is going to compete soon. Twitter is different, it's more like MySpace.

StarLink isn’t profitable, so that’s the kiss of death in a down market. Doesn’t matter how good the product is or how much people will pay if the replacement satellites to keep it running cost more.

StarLink is the right out of the book case for VC investment. One should expect several years before it gets profitable, and with very large initial investment. It's also very difficult to replicate, so whatever market there is there is a high likelihood for it to capture.

That means that it should be judged like a startup. Did it demonstrate a market? If so, it's valuable, if not, it's not. Current cash flow isn't very relevant.

Starlink is likely roughly break even launching on Falcon 9. The latest $137B valuation is likely predicated on Starship working as expected.

Still, when the president of a company tells that to his employees, you can't completely ignore his words, even if it's Elon Musk.

And that outage was the first of its kind in a long time and made a lot of news. It's not like this is usual in any way.

Why do we care so much though? Given the outsized influence Twitter has over politics and society, is it so bad if it shuts down and we have a few years reprieve? Musk has just lost an ungodly sum of money and has loans to repay. He’s not going to be running things for the benefit for humanity.

Musk needs to care because if his little site is not online when people want to post, they won't use it. I applaud that he doesn't though, makes it so much funnier watching this unfold.

Do you remember July 08,2022 when Rogers Communications went down for about 19 hours and services like 911 and Interac a payment service almost all Canadians use was interrupted? I guess it can happen to even the most critical of services.

It’s running well because the old guard built it so well. Musk threw them all out, so it will slowly but surely degrade. There was a reason Twitter had such a large workforce.

Is this the same twitter who almost died trying to scale when they started? They have been in worse spots. They will be fine and much of the old guard still works there.

So under what conditions would you consider a company overstaffed?

I would say you are overstaffed if A) you can't afford them and / or B) you've grown so fast, you don't know how to make use of the staff you have. Perhaps there is a C) which is a variant of B) the business model you are running only requires so much staff and you hit a ceiling in growth.

I would not claim to have enough information about twitter being overstaffed? They might have a lot of technical debt, which can cause the need for more people to run it. But that is not "overstaffing".

The question is also how overstaffing is measured. If adding another person reduces the average productivity, you might define that as the cut-off due to the declining ROI for every other person added. But you probably also want some level of redundancy (aka reducing the bus factor) in order to account for the ongoing global health situation and general churn.

Of course repeated waves of mass layoffs, demanding a pledge of allegiance to the company and revocation of all employee privileges at the threat of being fired and publicly throwing your employees repeatedly under the bus are also ways to reduce churn, so the bar for "overstaffing" might have actually lowered.

The tragedy of Twitter is that:

- they were absolutely undoubtedly overstaffed

- Musk's approach was absolutely undoubtedly insane shenanigans of a stupid rich cry-baby

Both are true at the same time

> they were absolutely undoubtedly overstaffed

Oh? What's your evidence here?

Look at FB's staff from back when they had comparable revenue and way more users.

I believe that 2013-14 FB was comparable in terms of revenue and users, and they had approximately half the staff.

So? WhatsApp had 35 developers and 350 million users. By that measure, every other company is grossly overstaffed.

Personally, I could argue that every social network is grossly understaffed by the measure of how lax they are about what they are supporting. E.g., in 2017, that company you say is more appropriately staffed contributed to a genocide that left tens of thousands dead: https://www.amnesty.org/en/latest/news/2022/09/myanmar-faceb...

WhatsApp wasn’t a social network with public posts of rich media. It’s trivial from an architectural perspective because there isn’t anything that needs to be sent to millions of users.

Oh? So you're saying a company with a similar, but not identical, product might have completely different infrastructure and staffing needs?

So we might not, in fact, be able to meaningfully compare Twitter, Facebook, Reddit, etc by headcount alone?

Exactly. It's very hard to know from the outside where the headcounts are or should be. E.g., Twitter had a ton of ML people. One could argue that was totally unnecessary, because Facebook didn't need that at date X. Or one could argue that they needed more of them, because it's the only way to wrangle the challenges of a large social network, and that hiring an ML person ends up being a net reduction in headcount because of the customer support/moderation people they replace.

> By that measure, every other company is grossly overstaffed.

Recent layoffs across a big chunk of the tech industry suggest this argument may have at least some truth in it.

> that company you say is more appropriately staffed contributed to a genocide that left tens of thousands dead

Contributing to a genocide has nothing to do with staffing levels and all to do with unhealthy incentives - the company makes its money on "engagement", and it turns out polarizing/divisive/violent/outrageous content generates a lot more of it than friends/family/cat pictures, so effectively moderating toxic content is actually against their bottom line and is only needed in extreme cases where the toxic content's negative PR impact outweighs what it will bring in "engagement".

This is incorrect:

> Contributing to a genocide has nothing to do with staffing levels

Unless we just want to outlaw online forums, any provider is going to need to track and moderate harmful content. Note that even something as small as HN has a full-time moderator. That's just part of the business model these days.

You're correct that moderation can run counter to short-term revenue gains. But it's much less clear that's true in the long term; as Facebook's reputation has worsened, its growth has slowed and its revenue has declined. Or on the other end of the scale, look at how the less restricted sites are doing. E.g., the 'chans are never going to move beyond a very small niche, and despite Twitter's troubles, nobody's expecting Gab to take the lead.

Personal anecdotes from any big company :)

There's a second layer to the tragedy. Also applicable to any big company:

- many teams are overstaffed for what they do/produce

- many teams are understaffed for what they do/produce

And this is also true at the same time.

Yeah, this is closer to the mark for me. It's been some years since I worked at Twitter, but if I had been suddenly made CEO, I wouldn't have gone on a mass firing spree. I would have tried to figure out how to more effectively use the people that Twitter had. As with many big companies, I suspect the problem is much more bad management than bad workers.

This is roughly my feeling, but I'm a little less sure of the absolutely/undoubtedly part. Someone below is asking for evidence but I don't think any one of us could produce evidence for or against this. It all boils down to either "Twitter had ~7500 employees and..."

1. ... that feels like a huge amount for a company that has basically a single product

2. ... that feels entirely appropriate for a company operating at their scale

What we've observed is that they've kinda managed to keep going despite Elon axing an enormous number of employees, but they've also had a few high-profile outages like this one today, the 2FA issue and 16hr Aus outage. I don't know if these are enough for either side to declare that they're correct.

One thing I would like to say - even though I had a feeling Twitter was probably overstaffed, I was in no way celebrating the redundancies. Similarly even though I'm not a fan of Elon Musk, I'm not delighting in all these little goofs he's doing and how it's impacting him personally - every little embarrassment he suffers and every dip in Tesla stock makes it more likely he'll just call it quits or fuck up Twitter more. For all its faults, I quite like Twitter and I don't want it to just disappear.

> It’s running well because the old guard built it so well.

Twitter is a shitshow technically. Sub-tweets don't nest right and change order in a arbitrary way.

The backend that handles the API has generally been relatively solid, but the official clients are a complete disaster.

This comment feels a bit out of place on a post about part of Twitter being down.

Currently it looks like 3rd party apps have been cut off, not the Twitter or the API being down. So I don't consider it out of place.

He haphazardly ordered turning off "bloated" microservices, including the one responsible for sending auth codes via SMS for 2FA... Learning through mistakes is a thing particularly for newbies, but a CEO should absolutely have a clue about not micromanaging, especially in areas he practically doesn't understand and after firing most people who do.

I'm unsure if it is running as good as before. I have noticed more issues over the past week than I ever have using Twit.

Particularly, a nasty bug that keeps the timeline refreshing while scrolling and is constantly moving content around on the screen.

Curious if anyone else has run into something similar.

> its up and running well

I would from personal experience disagree on the "well" part.

I'm suddenly following accounts I recently unfollowed again. Previously blocked accounts show up. The search doesn't find stuff it used to find anymore - posts that are definitely not deleted because I can usually find them via other references.

I have noticed that the explore page, and my timeline are way staler than they used to be. Tweets I have seen or even interacted with keep popping to the top all day, and trends on the explore page can be there for days at a time. It can take hours for major world events don't show up on the explore page for me sometimes.

If you buy a house and move into it does it suddenly break apart? If not, does the new owner get credit for it not breaking apart?

It’s working _in spite of_ the new owner, not because of.

If you’re going to argue against what “people” say, you’re always going to look like you’re the smart one and the person you’re disagreeing with is the stupid one.

Really don't know why this is downvoted. We had relentless articles like this [0] expecting the complete and total shutdown of Twitter, globally. That never happened and everyone knows that it was pure hysteria.

> Since Musk took over we haven't had all the downtime and disruption everyone kept claiming we would have. Big companies haven outages too, but so far the death of Twitter isn't in sight.

So because [0] did not happen most of the critics are setting a new standard for Twitter uptime to be 1000% globally 24/7, absolutely zero outages of any kind and never down for one person. If one outage somewhere happens, it is immediately the end of the world.

It wasn't the end of the world when Facebook, Instagram, Snapchat or TikTok had worldwide outages, but 1 regional outage on Twitter and it is somehow the end of times? Please.

The fact is, You are getting downvoted because Twitter is still up and running and many of the replies here cannot stand that predictions like this [0] did not come true and will point at any intermittent issue to show the 'Twitter is dying' narrative.

The death of Twitter has been greatly exaggerated. If you really wanted it to die, just ignore it. Simple.

[0] https://www.technologyreview.com/2022/11/08/1062886/heres-ho...

His comments on Taiwan were prompted comments in request to a question buried in some interview. He didn't rant about it. He hasn't ever mentioned it on Twitter. And how he was quoted was very different from what he actually said.

I said his comments. Not his tweets. I’m fully aware of what he said. It was all over the news in Taiwan. He pissed off a lot of people here.

He showed everyone how in bed he is with the CCP.

I think he showed he doesn't really care about Taiwan that much. It was an off-hand remark, not a seriously stated position.

Reply on news.ycombinator.com