Hacker News Reader

Comment by perihelions | Hacker News Reader

I don't understand the social structure inside a software company where this kind of thing can go from some intern's 3am idea to production, without passing many layers of gatekeepers, any one of which should have swiftly flagged this down. It's not that the string replacement was implemented wrongly (that too)—it's that they're touching, in any manner at all, one of the most obviously-sensitive UX things in their product. Without a commensurate amount of security review.

Like, in my imagination, within five minutes of anyone seeing this, a person with responsibility would have stepped in and said "No, you can't do this. And if you insist on doing this, here's five layers of audits and sign-offs that this needs to go through first, because the thing you're proposing is potentially really dangerous". Am I thinking it about it wrongly?

I cannot understand at all.

View on HN

> Like, in my imagination, within five minutes of anyone seeing this, a person with responsibility would have stepped in and said "No, you can't do this. And if you insist on doing this, here's five layers of audits and sign-offs that this needs to go through first, because the thing you're proposing is potentially really dangerous". Am I thinking it about it wrongly?

Which part of "anyone who is not a Musk yes-man has already been fired or quit" are you having trouble with?

It's worth remembering, there are two kinds of yes-men:

1) the sycophant who loves authoritarian institutions; the "true believers"

and

2) the young, brilliant visa holder who was the talk of his parents' social circle in Hyderabad three years ago when he graduated and was able to get on board at a household-name NorCal tech company, but who is now being abused by the employer who sponsors the thing that lets him stay in the US.

You'll always have type one; some humans simply love following a dolt. The second type is a result of our laws, and laws can be changed to keep people like Elon from taking advantage of workers.

The visa holder thing might have been true for maybe 3 months before and after the takeover. The job market was absolutely on fire at the time (early 2022). Anyone who wanted to leave should have been able to, especially considering they were good enough to be hired at Twitter.

> The visa holder thing might have been true for maybe 3 months before and after the takeover. The job market was absolutely on fire at the time (early 2022).

The Twitter takeover (and subsequent layoffs, ultimatum, etc) happened in late 2022.[0]

[0]: https://en.wikipedia.org/wiki/Acquisition_of_Twitter_by_Elon...

But it was announced in April. And it wasn't hard to predict what would happen post-acquisition.

I would argue laying off an insane chuck of the company was not predicable. Some? Yes happens all the time. What Elon did? No. But visa holders would probably be the first to go. But if you did manage to survive, no way you’re quitting in that job market and getting a new gig in a saturated market.

Visa holders wouldn't be the first to go. Not by a long shot.

1) They come from places that have much lower pay for software engineers. Seriously, people forget just how much more American tech companies pay their devs than just about anywhere else. That means that you can wave a smaller pay package in front of them and they'll be more likely to take it. Replacements from local labor markets are more likely to know what they're actually worth.

2) If they aren't making you 100% happy and they're at-will employees, you can hang their entire lives over their heads. Do you want to have to pack up everything in a few weeks and move back to your home country in shame after your families sacrificed so much to get you here? No? Sounds like you'd better sleep in that conference-room-turned-bedroom, then, and be ready to work 90 hour weeks until this company's in the black.

You're thinking of people who work at consulting body shops. Literally none of these things applied to visa holders at Twitter.

Go re-read all the comments here during the take over and layoff where people claimed it could not possibly take more then a handful of people to run such a simple site.

he fired a huge number of the staff and the site is still running, so how was that assumption not proven correct ?

I use twitter daily and the site is a shell of its former self. It's slow, prone to bugs, filled with bots, the amount of real users has cratered, user reports go nowhere, there's no support team, the ads are now bot accounts posting crap like "Today is a good day, be sure to make it advantageous", there are no new features besides previously in-flight projects pre-Musk, they've actually removed a lot of features (like Circles, block lists, etc), and much more. He took an otherwise functioning social media service and forced it into maintenance mode. He also fired all of the people that keep the user base alive so now it's flooded with bots (which he presumably likes so he can boast about engagement being up). So yes it's still around but it's dying and the skeleton crew he has left can't do anything.

In other words, he destroyed it.

The bot plague is atrocious. Like, there are tons of "keyword watcher" bots... write "onlyfans" and you'll get ~5-10 spambots in under half a minute, and for stuff involving popular politicians or political events (anything Russia, Ukraine, Israel, Palestine, Covid) you'll get Russian fake newspaper clones. On top of that come the "human bots" - write the name of infamous German youtuber "Drachenlord" and you'll get that vile hater bunch and it's just the same.

Honest question, why on earth are you on a fascist social media platform?

Honest answer: it's where my friend group's group chat lives. I miss a lot of conversation if I'm not in it. To be fair, I've blocked 235k+ Blue accounts so my experience is actually a lot better than most users.

The site was previously engineered such that an uninformed owner could literally start unplugging servers without it going down.

That said, outage frequency did rise following the mass layoffs.

https://www.nytimes.com/2023/02/28/technology/twitter-outage...

> he fired a huge number of the staff and the site is still running, so how was that assumption not proven correct ?

Why would you expect the website to stop running though? Keeping a site running with a smaller crew is easy amd baked in - all organizations with >10 engineers do this frequently, over the holidays or Lunar New year. What's harder is building new features at the same pace and quality as a larger engineering team.

Don't take this as a defense of what is a harebrained idea; but this kind of replacement should be easy to do correctly. You know; in such a way where only real twitter.com links are changed to x.com.

Honestly it is only somewhat surprising to me that no one noticed the error ahead of time. On the one hand, this is the type of mistake I do see in reviews from time to time... usually in the form of a regex that is not anchored to the start of the string, or perhaps it uses a non escaped period which of course means "any character" in regex. On the other hand, it is revealing about the kinds of controls in place that it got through.

even things that are easy (or maybe, especially things that are easy) can benefit from having a review to make sure you didn't miss something obvious.

nothing more dangerous than "oh, that's easy, let's push it straight to prod"

Example: one of the worst and most costly bugs I ever saw during my time in finance involved a code review [1] with the exchange:

"Looks good generally but this will serialize incorrectly if a date is ever negative."

<... pause while everyone digests that...>

"Why would a date ever be negative?"

Noone could think of a reason a date would ever be negative. The resulting code push cost millions in a single day.

[1] I was on the call because I was the SRE[2] responsible for doing the release

[2] We didn't call ourselves SREs because it was like 2001 and that terminology hadn't caught on yet. We called ourselves "Installmeisters" believe it or not.

How did it turn out that a date was negative, and how did it cost millions?

Iirc it turned out some 20+ year old code was relying on an undocumented/unknown behaviour of dates when they are negative to calculate cashflows on very old perpetual fixed income instruments[1]. If you have a codebase that's large enough and worked on by enough people, for every possible hack/undocumented feature you'll have at least one person who will rely on it in some way.

It cost millions, because the serialization corrupted the USD treasury curve when we wrote it to the database, which meant that even though the code was reverted quickly, for a while no derivatives (like literally no derivatives) in the whole of the trading part of a big IB could price (because essentially all derivatives were priced in dollars using some sort of discounted cash flow mechanism which relies on the USD treasury curve). No price means no risk, because how you calculate risk of a derivative is by finding the underlyers, shocking those underlyers a little bit and calculating the new price (essentially an empirical method of finding the derivative of price with respect to each of those things).

[1] https://en.wikipedia.org/wiki/Perpetual_bond

That's a good hypothesis with the non-escaped period! That someone wrote /twitter.com/ for the string substitution, which almost works, and then added a second one like /\w+.twitter.com/ for subdomains, which also seems to work, and would pass simple tests to check if it works or not. Matches everything it's supposed to; rejects most of the things it shouldn't.

You are right the period would line up with the issue we saw.

But also, I can’t help but to think even /\w+\.twitter\.com/ will match unexpected domains like foo.twitter.com.evil.ai. I already gave you the hint it needs to be anchored!

Well, that is the specific outcome Elon wanted when he laid off three-quarters of the company. He very clearly stated that he didn't see the value of the "trust and safety" teams that would have been the ones to flag something like this down.

This wouldn't have fallen within "trust and safety" purview. This should've been caught in even the most cursory of code reviews.

It seems this may have only affected the X on iOS App. [1] That greatly expands the range of possible causes. It also makes this quite odd in another way as well, because it suggests this was not a server side change.

[1] - https://news.ycombinator.com/item?id=39991312

Why wouldn't phishing fall under trust and safety? Perhaps not primarily - but it's definitely a trust and safety event.

They wouldn't "have been the ones to flag something like this down". It should've been caught long prior; the code shouldn't have been written, it shouldn't have survived code review, and it should've failed automated tests.

It's a trust and safety incident now, but it never should have been.

The trust and safety team was only responsible for censoring wrongthink

> I don't understand the social structure inside a software company where this kind of thing can go from some intern's 3am idea to production

This is what happens when you "cut the fat" and are left with an adversely-selected[1], skeleton crew of "hard core engineers." The site was never going to fail all at once, instead, it's a death by a thousand cuts and suboptimal engineering.

1. No disrespect to current Twitter engineers who can't leave easily, or believe in the mew mission. However those who survived layoffs but could leave have left.

Well, that’s because any org that has five layers of audits for this just has five layers of audits for everything and so rarely gets anything done. This is a clbuttic bug. It’s silly and damaging but easy to fix and move on with.

Not particularly different from Bluesky allowing one guy to own all of S3.

Well it starts with the bad idea probably coming from the top, Musk saying I'm tired of seeing twitter.com links change them all to look like x.com links, that plus his gutting of the company when he took over means there's less people around to be the person to say no you can't do this this way go back and start over (or not at all).

It's probably worse with Musk. His executive style seems to be, from his biography, ignore a thing for a while until he gets in a maniac phase and then over the shoulder manage a thing until it's done, regardless of time or context.

I can just take the scene of Musk being on a roof yelling at the crew to change how they install solar tiles late in the night and translate it to him berating a programmer in the office to make it look like x.com and not caring about the details.

This article was 5 hours old at the time I'm viewing it and the bug is supposedly already fixed per the article itself. So yea, seems like this was probably fixed within 5 minutes of anyone noticing.

Anyone who would tell Elon "this isn't a good idea" left or was fired.

> without passing many layers of gatekeepers

Given all the Twitter bugs and issues, it seems they have all been laid off.

>I don't understand the social structure

There is none. Stop trying to understand - it's a fool's errand.

>a person with responsibility would have stepped in

There are none left. They were either laid off, or they left before that could happen.

>"No, you can't do this"

... is the last thing anyone says to Elon Musk before getting fired.

> some intern's 3am idea

I think you mean "Elon Musks random demand".

I mean, I assume anyone who could get another job has already left, so they're probably running low on competent gatekeepers.

Reply on news.ycombinator.com