Twitter Is DDOSing Itself
sfba.social
sfba.social
Even more so when that person later loudly proclaims that they never made such a request, even when provided with written proof.
I can of course not say whether the people currently working at Twitter did warn that the recent measures could have such major side effects, but I would not be surprised in the slightest, considering their leadership's mode of operation.
Even as someone who very much detests what Twitter has become over the last few months and in fact did not like Twitter before the acquisition, partly due to short format making nuance impossible, but mostly for the effect Tweets easy embeddability had on reporting (3 Tweets from random people should not serve as the main basis for an article in my opinion), I must say, I feel very sorry for the people forced to work at that company under that management.
I worked in the games industry for a while, and came to understand how they could spend so much money and so much time, and yet release a game where even basic functionality was broken. It's exactly this sort of extreme schedule pressure that, ironically, makes a huge morass where changing one thing breaks 10 other things, so progress grinds to a halt.
Not necessarily. I’ve predicted bad outcomes for decisions in a few cases and been ignored but stuck around regardless. Mostly because I like my job and the goals of my organization even if it makes bad decisions.
Of course to remain productive and improve my influence in future decision making it is absolutely critical that when predictions come true, I do not go anywhere near an “I told you so mentality.”
Instead I do what I can to clean up the mess with a “how can I help?” attitude. And increasingly over time people take my opinions and analysis much much more seriously.
I wouldn’t say that’s the path everyone should take, especially because some work environments are just too toxic for any progress at all (I ran away, fast, from two jobs like that). And some people cherish having an entirely new type of challenge every few years instead of shepherding something through longer periods of time. All valid paths.
You have zero idea if that is true or not.
This is like a case study in what happens when you fire everyone except the sycophants and yes-men.
I only feel sorry for remaining non-yes-men twitter employees who might still be there because for whatever personal reasons they're in a precarious economic situation where they can't quit (H1B?) or are tied to the company for healthcare coverage (Thanks, America, for being the greatest country in the world) because they can't afford any other health insurance option.
What I maintain though is that most anyone still working on code at Twitter, regardless of their experience or overview of the code base, would strongly argue for testing and staging, which appear to go against current leadership's mode of operation, likely because of the time pressure you mentioned.
Not pushing such changes straight to production is a concept I feel anyone working at Twitter would subscribe to, yet has to painfully go against, lest they be led go.
The only difference between then and now is that there is a big personality at the top who now personifies everything Twitter does, especially if things go wrong, whereas before it was mainly just a faceless bureaucracy whose Trust and Safety lead at times had more visibility than the CEO.
The subtext is that Twitter changing hands also involved trimming a lot of the dead weight, particularly hitting the softer managerial/diversity/HR side. Now a lot of people are rooting for the site to fail because it has gotten too "bro-ey", as the era of trust-and-safety and $15k backchannel bluecheck deals has made way for free-speech and monthly subscriptions.
lol nobody would do this to solve this problem because it doesn't even remotely solve it or give the appearance of solving it, if anything it's guaranteed to make things go slower
Seems more like a thinly disguised attempt to force people to purchase subscriptions. Blue checkmarks and all.
Are you saying the engineers who are now at Twitter don’t have the right skills?
And never created sufficient testing.
But this is Scaling-101 stuff. It's not some super complex or unique system going wrong. At least according to the article, it's a classic case of bad retry logic leading to a death spiral.
Unless the home feed being down is simply a side effect - the service that fetches tweets being DDOS'd by other views in the app making numerous non authenticated calls.
But I was also thinking about this earlier today. These days, everybody is so quick to say "the software is easy, it's the community that's hard" - I've even said it myself a few times in the past few weeks, but I think that might be overstated.
Building good software is hard. Keeping it good is even harder. What does the codebase look like for Twitter's front-end at this point?
How many frameworks has the base functionality been ported through? How many quick pivots from product adding features, adjusting things squashed down the ability to address technical debt or even have functioning unit and regression testing?
The fact that this 1. Made it to production and 2. Was not noticed and rolled back immediately (like, in under 30 minutes) is extremely concerning (and obviously very embarrassing.) If I had private data stored on Twitter of ANY kind (like DMs that I don't want getting out - a messaging system rich, famous, and powerful people have been using like email for over a decade), at this point I would be trying to get that data removed however I could, or accept that there's a strong possibility there's going to be a huge data breach and all of the data will be leaked.
That would give the server side more control over the retrying logic (when the header is properly interpreted). I'm surprised Elon hasn't implemented this himself.
If you’ve never had to handle authorization in a particular area, it might have been safe to assume that any 4xx error should have been retried when the code was originally written and someone didn’t write them all out
Though it’s hard to know for sure what really went down. Could be a number of things. Including a lack of subject matter experts (Elon recently admitted to laying off some people they shouldn’t have).
If you use a listener, useEffect in react, to load data, it will start the request, track it is loading with a boolean, and then store the payload. That passes unit tests and QA.
If the listener doesn't check the error before starting the api request again, you have this infinite loop happen where the loading flag goes off and the payload is still null, so it just starts it again.
It's sloppy code, but its an unintentional side effect.
That usually gets them to sit up and reconsider.
Occasionally they sign the damned thing, and then it’s popcorn “I told you so” time.
But when a higher-up so clearly takes responsibility of something that is risky but deemed necessary, it provides for a lot better space to also respond to any potential problems that pop up, while not needing to suffer snarky comments in hindsight as a dev.
Of course something like this is easier to hate on ...
Will admit that seeing a billionaire go mental is funny. Petty I know.
In this case the horrible idea is being forced to push changes to production at a moments notice
As said, I don't want to speculate whether someone raised their voice on this being a likely outcome specifically, as not having testing prior to release is the much more obvious and significant issue here, driven by leadership.
Isn't that one of the DORA metrics?
The push to "make the content less user friendly" and push then to our app/logged in experience is similar to reddit's push and it doesn't really present an insurmountable challenge on the technical side.
These technical hurdles are temporary. Not the end of the world. The "only in our app/only logged in" requirements are bad on a philosophical level of user choice, not on the "this is terrible technically" side.
Attacking it on that note is just perpetuating the narrative where elonmuskmanbad (which is more political than consistent with the treatment others get) above more specific truths.
As an industry we need to stop perpetuating this idea that everyone is fully mobile and software engineers are some mythical creature that isn't also shackled to the same constraints the rest of the labor force is. It's a job and a lot of people are dependent on a steady income stream. Part of that is putting up with the least worst of the options they have reasonably available to them, or sometimes taking up worse options just because it balances other aspects out. Ultimately during certain times you may have more or less leverage to put up with or reject crap practices. SWEs tend to have a lot more leeway than say a retail worker of course so we can be a bit more demanding but we can't just say crud off, I don't deal with any BS, even in the best of times. Higher salaries give you some negotiation room as well as you can lower your TC expectations in terms of negotiating better working environments (i.e. taking lower rates but demanding things like less pressured schedules, make autonomy, etc.) and as the extreme sof the markets are high enough, even taking TC hits can still let you live comfortably (this is to contrast many other professions who don't have this flexibility).
Unless you've lived modestly and invested significantly or started some side venture to become financially independent of labor based income, you're ultimately giving up some degree of agency to employers. We like to pretend this isn't the case but it is the case for most people, even highly paid labor.
The second instance, I feel I can comment on, though allow me to leave out details to spare the people involved public embarrassment.
There, the changes, whilst annoying and born out of a complete misunderstanding of a core part of their alleged competency (Imagine a Botanist telling you leaves are always blue because that is the color of the sky), were not going to break anything, just look silly and create unnecessary, but compensated, work.
In that case, I also viewed the specific project finally launching as vitally important to our user base and wanted the results to go public for their benefit, so the decision was made to document and execute on their requests, so we could go live.
Of course, two weeks before our go live date, they changed their requests again ("leaves are not blue, they are violet because of the wave length of light") and had a hard time understanding that changes can have a knock on effect and some things are a bit more complex than Find and Replace. If I had been forced to make those changes at that point, I'd have packed my bags.
Simply said, when my work has the potential to benefit users and I know that arguing, even though I am correct, will lead to massive delays, I'd rather just put that silly request in writing and deal with these things after the users have received what has been worked on. Try to explain when someone is wrong, but if that doesn't work, finish the project, argue later.
Of course, if after the fact responsibility isn't taken, that finished project gets mentioned in my CV for the next employer.
In the case of Twitter, the new owner has thoroughly broken the advertising business and is trying to aggressively pursue a new version of the data business Twitter once had-- E.g. Google's Caffeine, which Twitter also eventually lost https://searchengineland.com/google-search-algorithm-change-... ... The statements about "too many scrapers" are almost certainly as illegitimate as the previous pre-acquisition statements about "too many bots."
The nature of business is that there's no judiciary or referee... the purpose of a business is to make money. Tech businesses just happen to hire lots of academically-oriented engineers who developed their skills in a different environment. It's possible to build a culture of "fairness" in a business, but at the end of the day even Google dropped "don't be evil."
Я, Unless your visa is sponsored by your employer.
Detaching myself emotionally from my employer was one of the best things I've ever done for my mental health. When I was young I got upset when "the company" made bad decisions. Now I feel no negative emotion about it, sometimes I laugh at them.
Doubt either care at this point.
This is something I don't get. They say that Twitter is now blocking embeds and I've seen an author respond by... including a screenshot of a tweet in their piece.
Except that's obviously better than embedding a tweet in the first place. It's better in every possible way. It's easier to write your article with an included image that you provide than to hotlink content from some other website. And when the tweet vanishes into the mists of history, your captured image of it will still be around, illustrating your article the way you were hoping the embedded tweet would do. There are so many articles out there with dead embedded tweets illustrating... something.
So, all that said... what was the effect that embeddability had on reporting? Why hasn't it been screenshots the whole time? What did easy embeddability enable that wasn't just as easy anyway?
Forced? You know slavery was abolished a long time ago no?
What I’m more surprised about is how gum and shoestring the twitter engineering is now a days. They put in no emphasis on doing deep divides into the code base and instead opt to do the simplest shortest fix. And it causes problems.
These are the hardcore programmers of Twitter 2.0, aren't they?
I have seen similar bugs in the systems I oversee because network libraries love to retry requests without sane limitations by default. But I never saw them make our rate limiters sweat. It's slightly more annoying when they hit an API which actually does some expensive work before returning an error but that's why we have rate limits on all public endpoints.
I also guess that the webapp is the least of Twitters traffic and the native apps probably don't have this problem.
Seems like either my quota reset or they changed the policy because I’m able to access the site again.
I agree that this is probably not the bug at the root of it all. But I also don't believe the story that Musk is selling for why he's in effect shutting down the site. But both could be true and I'm still thinking about other potential reasons, a complete waste of my time, but it's a weird mental honeypot.
The book "Nothing is true and everything is possible" describes Putin's use of misinformation to maintain control of the populace and eliminate democratic types of politics, but it really feels like it applies here too. There will always be Musk fanbois who will parrot whatever he wants them to say, but most know it's just self-serving BS. And anybody trying to get to the root of everything gets easily sidetracked into narratives that feel right but have zero data backing them, like this bug.
Anyway, highly recommend this book if you want to see a likely path for the future of the US:
https://en.wikipedia.org/wiki/Nothing_Is_True_and_Everything...
"Nothing is an absolute reality, all is permitted"
The media is flawed. Journalists are flawed. The Verge in particular has some atrocious coverage of tech.
But the problem with Elon is that in his world, we must rely on him for true information. As if the direct source is inherently true. He’s just as full of it as anyone else with interests.
My favorite is when things are claimed to be impossible, that Elon's lying because the thing is impossible, but then it happens anyway.
Where Elon gets in trouble is he's wildly over-optimistic on a few things, such as AI. He predicted an AGI would take over in 5 years around 2015 or so (so we're 3 years off), and I think he really believed it. That's why he's always claiming things like self-driving, and doing it without sensors or whatever. His paranoia of AGI and his over-confidence on self-driving have exactly the same root cause (believing AI will conquer all). Elon has had so many instances of overcoming status quo expert predictions (whether on solar energy, battery-electric vehicles, reusable launch vehicles, or whathaveyou), that I think it makes him increasingly unable to very effectively listen to experts.
And he's also incredibly gullible and easily taken in by all sorts of scammers, including rightists and just plain sycophants telling him what they think he wants to hear. Which is increasingly what he's left with as everyone else who is sick of his bull has left.
Is it a lie if you don't know or even care if it's true or what it means? That's where we are with Elon Musk.
I will note that the few times I investigated claims of Elon lies they were not proper lies, either being misunderstood, misleading (which IS unethical, don't get me wrong), of indeterminate truth value (he said, she said type stuff), delusional optimism or actually true.
Like journalists, Musk rarely outright knowingly makes literally false statements, but this does not mean you should take what he says at face value.
Infact it got so bad because of all those retries at multiple levels from upstream callers that requests were essentially timing out at the TCP buffer/queue before they could be processed by the application.
Don’t know if the Twitter homepage backend is at similar scale.
If IPs or IP ranges get really annoying we block them on the network level.
Big public sites like Twitter obviously need to have this technology. Due to their political content they probably also need sophisticated DDoS protection.
Judging from the screenshot, a huge amount of GET /TweetDetail is generated which triggers some rate limiting, as shown by the 429.
If this is indeed due to the recent decision to enforce authentication for all API calls, it means the curlprit may actually be the API gateway or something similar downstream.
Also, this behavior seem to never stop, which isn't what one would expect from an exponential backoff retry.
I don't claim to be a better engineer than the folks working at Twitter, but it is interesting to see something like this in the wild, all Musk-related considerations aside.
Like several times in different roles.
People do it, exponential backoff is everywhere in your stack, but it doesn’t end up in your application layer until you have enough traffic that you actually have to manage throughout.
The way I understand it, DDoS is not caused by enforced authentication - enforced authentication is just a temporary measure against DDoS.
> Twitter’s Google Cloud contract dates back to 2018.
https://www.engadget.com/twitter-has-supposedly-started-payi...
Oh... this explains all this fiasco :facepalm:
That's from a week ago.
Gcloud only supports batch jobs and data analysis.
[0] https://twitter.com/elonmusk/status/1674865731136020505
> Temporary emergency measure. We were getting data pillaged so much that it was degrading service for normal users!
[1] https://twitter.com/elonmusk/status/1674942336583757825
> This will be unlocked shortly. Per my earlier post, drastic & immediate action was necessary due to EXTREME levels of data scraping.
> Almost every company doing AI, from startups to some of the biggest corporations on Earth, was scraping vast amounts of data.
> It is rather galling to have to bring large numbers of servers online on an emergency basis just to facilitate some AI startup’s outrageous valuation.
Surely there are some colossal archives of all tweets before whatever date floating around that those startups are using instead.
Dumb but obvious consequence
I don't doubt that Twitter saw a massive increase in traffic recently, but I feel at least somewhat confident that it's mostly self-inflicted on Twitter's part.
Hahahahaha! Man, Parag Agrawal and his team knew exactly what they were doing. A fool and his money are soon parted, indeed.
No Twitter, no Arab Spring. No live updates of disasters, no need to shutdown the internet to prevent dissenting voices from spreading.
https://twitter.com/elonmusk/status/1675214274627530754 and self-reply
Twitter links are now poisoned, and basically useless.
I still remember watching podcasts, reading blogposts, hearing prominent VC's like Elad talking about etc.. Now everbody silent because Elon is the baba yaga...
Twitter was not bad because of its tech. Musk made it worse.
You have multiple clients (android, ios, web) and a whole cluster of microservices calling each other. The microservices calling each other will easily use up more requests.
IMO the likely culprit is rate limits being triggered by a huge increase in scraping caused by the new api pricing.
So, there may be a much simpler explanation for why their new rate limits on regular users to can-barely-scroll levels, and how that has all sorts of unintended consequences they weren't ready for
It seems an outlandish claim, but then again Muskified Twitter has previous form for this kind of thing with that time when they self-derailed by locking themselves out of their own api, right?[1]
You can’t debug and root cause if you do. So you end up adding hacky point fixes
The root cause is business & feature experimentation at scale with a tight runway & no executive oversight.
But also, yes, I use it to observe the latest meltdown; why shouldn’t I?
Those who do care are very loud (on both platforms) but they might be a minority in practice.
Their default of “just go ham on that API” feels like the same footgun of “by default this Humongous Database is wide open.”
The curse of IT Operations: if you do everything right, management grifters never think you did anything at all. Why do I need you? Everything works fine!
Most Internet platforms are put together by popsicle sticks and bubble gum. If Reddit didn't have 2000 full time employees babysitting that steaming POS, it would be offline by the end of the week. But Twitter was like a chicken running around with its head chopped off, and it didn't stop running for 9 months. It's a testament to how incredible those engineers were.
This collapse at Twitter should have happened by January. It's a real shame it didn't. How many thousands of tech workers were laid off because Musk's fellow parasitic oligarchs saw Twitter running "fine" after the lobotomy, and followed him off the cliff like a bunch of lemmings? How many billions of dollars has the economy lost because of this one despicable man?
Elon Musk is a fraud. He is not an engineer. He's a lazy bum who mooched his whole life off his groomer daddy's apartheid emerald money. He would be nothing without that disgusting, incestuous old man. Despite his enormous financial privilege, Elmo was too lazy to enter the US legally. He was in the US for many years as an illegal immigrant. He only bothered to get off his lazy ass and finish his Economics degree because the risk of his deportation was becoming a real problem for X.com. That crooked philistine doesn't know the first thing about hard work, engineering, or finance. He's a damn good conman, he's good at shitposting, and that's about it. He has no other skills. He is not an engineer. He has no STEM degree. His only real accomplishment in life is proving that you can, in fact, spend billions of dollars in one lifetime, if you simply buy a bird app, run it into the ground, and salt the earth behind it.
Musk has a bachelor's degree in physics from the University of Pennsylvania: https://www.snopes.com/fact-check/musk-physics-degree/.
Further, while I share some criticism of the man, many very technical people from companies such as SpaceX and Tesla have come forward with public comments, praising him for "truly grasping the engineering" and "being involved in every technical design decision". Make of these what you will:
Kevin Watson, Falcon 9 avionics:
Elon is brilliant. He’s involved in just about everything. He understands everything. If he asks you a question, you learn very quickly not to go give him a gut reaction.
He wants answers that get down to the fundamental laws of physics. One thing he understands really well is the physics of the rockets. He understands that like nobody else. The stuff I have seen him do in his head is crazy.
He can get in discussions about flying a satellite and whether we can make the right orbit and deliver Dragon at the same time and solve all these equations in real time. It’s amazing to watch the amount of knowledge he has accumulated over the years.
Tom Mueller, SpaceX founding employee:
We’ll have, you know, a group of people sitting in a room, making a key decision. And everybody in that room will say, you know, basically, “We need to turn left,” and Elon will say “No, we’re gonna turn right.” You know, to put it in a metaphor. And that’s how he thinks. He’s like, “You guys are taking the easy way out; we need to take the hard way.”*
And, uh, I’ve seen that hurt us before, I’ve seen that fail, but I’ve also seen— where nobody thought it would work— it was the right decision. It was the harder way to do it, but in the end, it was the right thing.*
Garrett Reisman, engineer and former NASA astronaut:
What's really remarkable to me is the breadth of his knowledge. I mean I've met a lot of super super smart people but they're usually super super smart on one thing and he's able to have conversations with our top engineers about the software, and the most arcane aspects of that and then he'll turn to our manufacturing engineers and have discussions about some really esoteric welding process for some crazy alloy and he'll just go back and forth and his ability to do that across the different technologies that go into rockets cars and everything else he does.
Josh Boehm, former Head of Software QA at SpaceX:
Elon is both the Chief Executive Officer and Chief Technology Officer of SpaceX, so of course he does more than just ‘some very technical work’. He is integrally involved in the actual design and engineering of the rocket, and at least touches every other aspect of the business (but I would say the former takes up much more of his mental real estate). Elon is an engineer at heart, and that’s where and how he works best.
Along with many others.
But that being said, a loop of users requesting access from Twitter's system is a thing that would burden Twitter's system.
Requesting access and denying access isn't free. You can definitely DDOS your own website by having all of your users repeatedly request access to it in a loop.
It seems that this statement can lead to different conclusions based on one's biases for or against Musk / freedom of speech:
1) The code base is a mess because the smaller, current team don't know what they're doing and/or management (i.e. Musk) is pushing them too hard and making poor decisions
2) The code base is a mess because the far greater number of engineers there before didn't know what they were doing and/or management weren't pushing them hard enough and were making poor decisions.
Both could be true but I'm going with something the more complex view that it's a little bit of everything, and that Twitter does seem to be moving in a positive direction overall.
Everyone has their horror story and extreme examples, but for any mildly profitable company that went through a decade or more of operation, I think it's kinda of a meaningless statement.
Man, we do need a public square.
It's comical at this point.
For the to;dw crowd: he makes the case that Twitter is by far the most useful social network to government agencies, because the graph is mostly public (unlike Facebook) and all users are both consumers and creators (unlike YouTube). Thus, it is being used by “researchers” and state departments to build what he calls “the Death Star of AI censorship engines,” and these moves to rate limit will cripple those efforts.
Barring that, turning off server side throttling or atleast making it less aggressive to slow the retry storm seems the most reasonable.
So the result wasn't an outage, it was a radical reduction in functionality.
I think that still qualifies as a self-ddos.
Maybe not
> if twitter isn't going down
I disagree. Ddos is a type of attack, not the result of an attack. If they're hitting their services way too many times in a distributed fashion, it's a ddos regardless of how it was handled.
Similarly, would you say this[0] wasn't a ddos because it was mitigated? I think not.
Funny things is Twitter views are hiher than ever and it will be fixed quickly.
(No reason to make a big deal out of it) But I think tests and refactor of the codebase will be necessary to increase their speed of innovation.
Its a hive of misinformation, disinformation and toxicity. Its succinct I guess, but nothing is eloquent or descriptive because of the character limit. And its full of repetitive "filler" information.
Who wants that in a foundational LLM dataset?
Maybe its OK for finding labeled images... But that still seems kidna iffy.
Now, this has been severely degraded by the changes that Musk has made. The spam in direct messages is off the charts now, whereas in the past I would get maybe a spam per year. And when one of my areas of interest has a post that gets popular, I have to scroll past all the insipid clout-chasing replies from blue check marks which get floated to the top of replies in an attempt to reward some of the worst people on the internet. Also the long form tweets that need to be expanded are a big deflation of user experience, as reading and replying to those are suboptimal compared to a tweet thread.
But this is also the general internet: 99% spam plus 1% quality. And the quality of the 1% of good Twitter is some of the very best of timer material out there.
And since LLMs have been trained on this same mix... they seem to be mostly good at filtering. But they do lie an awful lot.
Or maybe you want to get an aggregate idea of what people are currently talking about in the world, stuff that doesn't rise to the level of capital-n News. There aren't a lot of alternatives for that.
Twitter is great for examples of that, and the toxicity and disinformation doesn't get in the way.
Conversely, a training set doesn't need to be up to date to be useful for that.
I don't know if anyone really was trying to scrape it (examples of Musk disagreeing with his own engineers come to mind), but I assume it's possible, and given the quality of code ChatGPT spits out I can easily believe a really bad scraper has been produced by someone who thought they could do without hiring a software developer. If so, they might think they can get hot stock tips or forewarning of a pandemic from which emoji people post or something — not really what an LLM is for, but loads of people (even here!) conflate all the different kinds of AI into one thing.
I mean as far as uses for LLMs go that seems to me a pretty realistic one. Mass quick propaganda with little effort. Go for immediate impact, doesn't matter if people look deeper, you're just looking to get a swell of emotional reactions.
Maybe... if you build a LLM scrapping for the lulz?
I'll add a dedicated microservice
I've now two scalability problems
Talents left twitter already, I wouldn't be surprised if the ones that took over are the ones who come from the intelligence industry, as opposed to the tech industry
I can't scale this more
I'll add a microservice
I can't scale these moreI don't think you understand what a "tyrant" is. Companies aren't democracies.
Yeah, gonna keep using it.
The webdev or JavaScript mentality of if it breaks, don't worry, we can fix it live strikes again.
https://news.ycombinator.com/item?id=36561808
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure
https://blog.twitter.com/engineering/en_us/topics/infrastruc...
429 requests handled at the load balancer are extremely light weight
Sidekiq falling over is a big one. See: <https://docs.joinmastodon.org/admin/scaling/> and <https://nora.codes/post/scaling-mastodon-in-the-face-of-an-e...>
(I have to email my own admin every few months to ckeck if things are OK.)
And during the October Revolution as hoardes arrived from birdland, things got ssssslllloooowwww globally.
Worked out eventually, but it took a bit.
Individual instances also tend to run into scaling issues, with Jerry Bell's Infosec.Exchange coming to mind. (Mostly because Jerry's discussed this a bit.) And of course individual instances can be shut down or fail in various ways. I've migrated several times myself.
I will say that most of the time things seem fine, and it's exceptionally rare for there to be truly Fediverse-wide issues.
(I've been on Mastodon / the Fediverse since 2017, for the most part quite actively.)
And the interface is not trying to assault me. It loads quick.
...Seems like a better product than Twitter for a public feed.
I prefer Twitter in the sense that it's more laissez faire in terms of what kinds of speech get you banned.
It isn't quite as decisive as a submarine imploding, and ceasing to exist, but it has turned into a brightly burning tirefire.
Twitter wasn’t healthy before Musk bought it. It wasn’t a thriving business, it was a very old, very large startup still struggling to find market fit and loosing a lot of money.
Also, it wasn’t a thriving product. It was stagnant.
Since Twitter was purchased, the amount of features they have shipped has been impressive. They’ve shipped a lot of features and extended the platform a lot. To your point they have also done this with far less engineers than before.
Regarding any downtime, everyone has downtime. Google, Amazon, Meta… the best of the best still have it regardless of money or manpower.
Considering what that team has done with less resources, I think the achievement still pretty good. What do you think?
https://news.ycombinator.com/item?id=36561808
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS.
https://blog.twitter.com/engineering/en_us/topics/infrastruc...
it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"
I tried to point this out below. It's not even a DDoS, it's fake news.
https://news.ycombinator.com/item?id=36561808
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS.
https://blog.twitter.com/engineering/en_us/topics/infrastruc...
it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"
That's a very good question.
https://news.ycombinator.com/item?id=36561808
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS.
https://blog.twitter.com/engineering/en_us/topics/infrastruc...
it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"
https://news.ycombinator.com/item?id=36561808
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure
https://blog.twitter.com/engineering/en_us/topics/infrastruc...
They had to change it to "Move Fast and Build Stable Infra" for a reason.
Reporting it publicly this way can also be a favor to both, as it gives them a fairly malleable narrative for reversing course, or directing responsibility, or both. Even if it’s simultaneously embarrassing to have it out there. And it’s not like either Musk or Twitter is a stranger to embarrassment, or particularly shy about courting it.
Like: we have the sources and you could detox you model if you pay for it.
Well, for one, this sounds just like yet another Elon-hater's comment. In _most_ cases, Elon's businesses are fairly successful, so a sarcastic "Elon's latest genius innovation" doesn't belong here.
For second, how do you know the idea came directly from Elon? It could easily be the result of a brainstorming session led by a product team.
Elon is not writing Twitter's code. He is not a Software Engineer at Twitter. Even if the idea came from him, why is he being attacked for a bad deployment?
And lastly, why do people think that the change is bad? There are plenty of web sites that do not allow you to view their content without creating an account first. Is Twitter not allowed to experiment? Why not? I would appreciate hearing well-thought-out arguments that explain the potential negative impact on the business.
Because it's quite literally one of the things he's responsible for at Twitter, by his own choice and description of his role (https://twitter.com/elonmusk/status/1656748197308674048).
(B) Are you familiar with the expression “the buck stops here”?