Decision Fatigue
antipope.org
antipope.org
We can revisit this prediction in a few weeks to get a sense of how sound this author’s worldview is.
Musk is embarrassing as always but he's right about one thing - engineers are costly, and for years now, the problem with Twitter as a business has been the huge outlay for engineering talent. And for what? No effect I can discern; the platform is clearly not a force for good in the world, first of all, but even looking at it strictly as a product, the product in fact gets worse every year and they're constantly discontinuing functionalities. So the whole "what has your code accomplished" show, it's inept but I know where it's coming from.
Kindof true. Platforms need a ton of maintenance - but usually the fires needing to be put out are because of other human changes.
Stuff like disk space, certificate expiration etc etc will bring down a platform without maintenance though.
* Apply security patches - there's thousands of dependencies.
* Manage hardware/cloud resources according to volume of data you need to handle.
* React to the changes in operating systems/browsers you run on.
* Fix bugs - they are there, because simply there's not enough time, money, and need to write "perfect" software. Competition will not do that and will beat you to the market.
There's no internet connected software, that can "just run", because world around is changing and you must catch up.
Fells a bit weird to explain that on HN :-|
We'll see what happens now, when there seems to be a drought.
I don't work at Twitter but these things happen at every business.
Twitter's raison d'être is allowing folks with millions of follows to see their posts in near real time, then monetize it with ads. That means analytics, billing, and more. From pretty basic requirements you can create a huge amount of work, especially when you're managing your own infrastructure.
I'm reminded of when, out of sheer curiosity, I once skimmed the proof for Fermat's last theorem by Andrew Willes. It was a monstrous work of which I could understand very little, but that clearly couldn't have been made by a team of a hundred mathematicians working together.
I also don't disagree that some problems are self-inflicted. Large organizations breed large organization problems. But large organizations usually grow from seemingly simple requirements and compliance efforts.
My team certainly got a lot more done before we required tons of process, safety checks, data storage requirements, etc., where we now spend a non-trivial amount of HC as overhead and avoid building other things entirely.
> I'm not talking out of my ass, I have been working on scalable systems for more than a decade.
Ditto, friend. I currently work in an organization that builds a billion user product with far fewer than Twitter does. However, we also depend on a battalion of internal services, and other shared resources (HR, recruiting, PR, finance, ...). Twitter maintains their own datacenters, are among the few users of Aurora and Mesos at this point, and purpose-builds a lot of their infra for requirements we probably don't have a great grasp on.
Twitter, like many products, is like an iceberg: the majority of the complexity is underneath.
For Twitter though I'm thinking though that they should have put a lot more effort in simplifying the systems/processes/data they store. This might even mean pushing to remove some business requirements that a PM might have introduced just for a promotion to reduce software complexity.
If you don't reward people for refactoring and removing complexity, I don't think you can just blame it all on organic growth.
Security patches... so 10 full-time engineers ought to take care of that?
"Thousands of dependencies" was a mistake. The ideal is minimal complexity.
Any browser or OS decides to commit a breaking change (rare), you bring in contractors.
Scaling up and down (and out and in) is handled for you automatically by cloud services now without anyone lifting a finger. "Fells" weird to be explaining that on HN.
Bottom line, you don't need thousands of engineers to keep Twitter running for "the next few days to week or two (at most)."
When has the world NOT been changing? The leadership of Twitter changed also.
the lack of autoscaling at twitter, imo, is worrisome as it sort of hints that the system might not be as mature in the self-healing department as compared to other large tech companies. time will tell.
you are right though, you don't need thousand of engineers to keep twitter running a few weeks. you also don't need a driver for a car driving a straight highway (until it bends). how many engineers are needed for maintenance? who knows. I don't think we have answers, we just have what elon decided.
disclaimer: facts based on twitter threads/online reading, not a twitter employee or anyone with first-hand inside knowledge.
On one hand I totally get your arguments, it's what I would expect my engineering team to say and they are often right.
On the other hand I am watching this space utterly intrigued if somehow he manages to pull it off.
If he does it might say more about the engineering team left than him though.
I would encourage you to read about “site reliability engineers” and the work that they do, in case you haven’t heard of them.
the fact aside that bug-free "just a bunch of software" of the size that serves half a billion people has yet to be invented. All software has the nasty habit of running on hardware and real infrastructure and that at the very least needs some serious maintenance.
If your standard for proper engineering is that it runs without human intervention at all I can't think of any complex system in the world that fits that description.
Do you have additional information about that you can share (maybe an announcement regarding this made by AWS)?
[1]: https://perspectives.mvdirona.com/2021/11/xen-on-nitro-aws-n...
A few examples:
- security holes discovered in the OS requires an emergency update, which requires lots of testing because updating the OS has a bad tendency of breaking even stable software (I've encountered that issue a few weeks ago);
- security holes discovered in third-party libraries/frameworks/..., which requires updating, re-testing, re-releasing stuff – and fixing whatever breaks because of undocumented changes in said library/framework/...;
- security holes discovered in your own code, of course, which also requires updating, reviewing, re-testing, re-releasing;
- monitoring your stack for misbehaviour, which could indicate a software problem (bug? license expired? SSL key expired?), a hardware problem, a resource problem (disk full?), an attack, or in the case of Twitter, one of your clients (the companies that build ads) misusing your tools and attacking you by accident;
- ... and once the misbehavior is detected, actually investigating and fixing the issue.
Sure, you can hop along for a while without anybody to handle these cases. But how long? Keeping in mind that Twitter is a high-value target for state-sponsored attackers (among others) all over the world, so any weakness will be probed and exploited mercilessly.
If Twitter is failing that easily, these engineers deserve to be laid off
Does your engineering department have a solution to all these problems that does not require human beings? Or are we talking of different things?
Twitter is a top-5 near realtime web property. It has an enormous attack surface across nearly the entire spectrum of cybersecurity threats. It's hosted at least in part on prem. Swathes of it are programmed in Scala, which is a minority language that is hard to hire for. It makes money through ads, which involves significant operational complexity at scale. Its recommendations and trending features are built on machine learning models which themselves require data engineering pipelines to deliver continuous updates. It's been in continuous development for 16 years, with an accumulation of accidental complexity (aka tech debt) in addition to the ever-expanding essential complexity inherent in its functionality.
Those of us who have taken an oncall shift in the world's most complex distributed systems environments are just shaking our heads at some of the drivel in these comments.
For instance this is why there needs to be a multi tiered (culminating with HN front page exposure) process for fixing automated account cancellations and app store rejections.
Sometimes you have automatic recovery. Sometimes you don't.
"We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness."
https://www.archives.gov/founding-docs/declaration-transcrip...
Or, alternatively:
"All human beings are born free and equal in dignity and rights." (from Article 1)
https://www.un.org/en/about-us/universal-declaration-of-huma...
The rights I appear to lose were taken away from me by a society.
I actually don't ever forget that.
There is also just frankly more traffic to serve because of mobile devices, so more things break at scale.
Other things which aren't necessarily needed to technically keep the code running, but without which you won't have an operating service:
* Sales and account management teams to keep and grow advertiser revenue, and all the support services around them (marketing, integration, etc)
* Moderation teams to keep illegal or dangerous content off the platform, and implement your moderation policy which you need for many reasons including appeal to advertisers
* Abuse teams to respond to the latest DoS / hacking / impersonation / harrassment / etc.
* Legal and ops teams to understand and implement compliance with laws, regulatory filings, etc across 150+ countries, and in some countries like the US, individual state variations.
You tell me how many people you can do this with at the scale of Twitter. I think > 1000 people are a skeleton staff just to keep the lights on.
I can write software that runs today. I don't know about tomorrow. Disks crash, other services crash, networks go down. No matter how competent your Byzantine Generals are, sometimes you have to recover data that was in-flight during a crash because disks and networks don't just crash, they slow down and they lose packets and sectors.
Add on security patches and changes in architecture (I'm looking at some Java 1.3 code that runs INSIDE an Oracle DB right now, ask me how I feel about not updating your architecture). You may say that architecture doesn't change that fast. If the business is successful, it will happen eventually.
That’s a bit like asking “why does any large internet service need infrastructure maintenance?”
If you can’t immediately see the answer in twitter’s case, then you either haven’t yet applied your knowledge of how large cloud services function and need maintenance to keep from cascading failures, or you don’t have that knowledge in which case the twitter case can’t be answered without explaining first the general case of how any large software system needs regular oversight. But then, I think that even if you can’t think of an answer for Twitter, you can probably figure out a little of the answer for the general question.
You can start with asking yourself another question: why would Twitter (or any very large software project) pay $Millions for maintenance that isn’t required? Is there collective delusion on the topic? Or has twitter hit upon the absolute perfectly engineered platform ever? If the answer to either of those is “no” then a twitter system failure is only a matter of time.
How much time is a matter of some debate. The first time something hits a quota or something of that sort that requires a human input to ensure automatic allocations don’t accidentally get insane could be a pebble. Or maybe the first pebble is a minor unpatched security flaw, but twitter is a very sweet high profile target… whatever the first pebble is, it makes a slightly worse issue or multiple additional pebbles much easier, and you enter cascading failure. They generally happen slowly relative to the proportion of a total collapse, and then very very quickly.
So another question to ask is how much of twitter’s infra teams were pure bloat, how much were excess capacity necessary to cover average amounts of employee vacation/sick/leave that goes on at any time, etc, and how elastic the remaining staff can be (putting in extra hours, working in areas that are secondary skills because no one else is available) and for how long they’re willing to do it before burnout or better job offers with less stress and overtime, whether staff losses have already hit a critical mass to make this inevitable…
But I think this gives a general sense of why it can’t tick over forever on its own, or at least some inroads into thinking about the answer.
Paragraph 1: There's lots of news
2: US Conservatives [*], and other authoritarians around the world identify with and support Putin. Further they identify with and support fossil energy resource extraction (PREE).
3. PREE are afraid of renewables.
4. PREE + Authoritarians are doing propaganda
5. Authoritarians don't want young adults to vote.
6. The young adults are not in favor of authoritarianism.
7; The old authoritarians will die.
Eight - Elon is in favor of disruptive change.
9ine ~ The disruptive change is going to have consequences.
10: No one likes Twitter.
E11even: Bitcoin is OK, right guys? (sarcasm)
* Not all US Conservatives are authoritarian, just the ones in charge right now. If you are a US Conservative reading this, then you are not an authoritarian, unless you wish to be identified as such.
All that said, I still find this to be the kind of thing that probably feels very tight and compelling if you already believe it (and perhaps if you already know what "PREE" stands for), but for someone on the outside (like me) it just seems like a bunch of statements with broad mood affiliation and then the blog post ends.
Author discredits themselves here. Big oil rules half the world. They are terrified of nothing. It shows a deep misunderstanding of the way the world runs or perhaps more accurately how it is controlled.
There’s are interesting splits here, too: for the 20th century the interests of the oil and car companies overlapped but now they’ve diverged with EVs, and there’s even a market opportunity for EV makers to turn on their former partners trying to gain the environmentally conscious market. I think the fossil fuel industry leaders are keenly aware of those inflection points where partners will leave them.
https://bestpracticeenergy.com/2020/08/26/energy101-electric...
Your linked article refers to the US only, but from that article, wind was 7% of power generation and solar was 2% in 2019.
By comparison, in the first six months of 2022 the equivalent number for wind is 11.5% and for solar is 5%: https://electrek.co/2022/08/25/us-renewables-first-half-2022...
So wind+solar have gone from 9% to 16.5% of US electricity production in 2.5 years? That is a fairly remarkable growth rate, assuming I am not misinterpreting the data somehow.
Look to Australia where renewables are being installed at very high rates.
There are many areas where they are dominating the power mix for large portions of the day.
As one of many reasons nuclear won't help much. There's not enough uranium to replace coal and fossil fuel gas.
I am old, and admittedly, my oldest grandchild cares more about Twitter than I do.
Maybe one of you here can enlighten me. Why does Musk get so much attention? Why is Twitter so important? Perhaps this is incredibly ignorant to say, but why not let people who want to use Twitter use it, and those that don’t, don’t use it?
I keep hearing about freedom of speech, toxicity on social media. Is it unreasonable for people who don’t like Twitter to just not use it anymore?
I feel the loudest voices are anti Twitter. Or they hate Musk. And Elon Musk seems to rather enjoy trolling these people.
Mainly, it feels like Twitter and FB have somehow become the defining issues of our time. Maybe I’m just stupid, but I don’t understand why. Why this obsession with Musk?
I’ve gone through layoffs several times in my career. As has my older son, as has my sister, my wife. So Musk bought a company, took it private, and fired a ton of folks. Has corporate America never had layoffs? Have we never had layoffs where honest, hard working people were needlessly fired, because of market conditions, short sightedness and greed on part of their business execs, or some combination of both?
I worry that this country is hyper addicted to needless drama, and it’s self imposed and obfuscating bigger challenges we have as a society.
I saw a Twitter conversation between Democrat and Republican House members. Absolutely shameful how these people hurl insults at each other, try to respond with sly, “witty” insults. Maybe I’m the one being the drama queen, but it feels like Rome is starting to burn and we have no one to blame this time but ourselves.
End of rant from this old goat.
As far as the members of Congress go, sly insults is an upgrade from a lot of what has happened the last few years.
And we have a tendency to remember politics as being gentler than it was. Check out this absolutely childish exchange from the 1980s: https://www.washingtonpost.com/archive/lifestyle/1985/03/06/...
It's easy to dig up any number of these, which used to be buried in the back pages of a newspaper rather than playing out and being memorialized on the web.
I see it as an established group who mostly benefited from Twitter (think journalists and traditional media who had great reach and influence through the platform) fearing that their power will erode under the new regime.
This is already playing out. Anyone who pays a small fee can be “verified” (a privilege previously reserved for the elect few), opposition/critical voices are returning from banishment, and Twitter is reconsidering which people/tweets/views are amplified or suppressed.
The reason why Musk/Twitter are “so important” is because those who stand to lose want us to believe so.
> I worry that this country is hyper addicted to needless drama, and it’s self imposed and obfuscating bigger challenges we have as a society.
Agreed. There are players and fans. The players (Musk, media) have stakes that matter. The fans (you, me, anyone who roots for one side or the other) may have ideological stakes, but it’s mostly tabloids for nerds.
I'd venture an alternative answer.
> Why does Musk get so much attention?
Musk is considered the wealthiest person in the world. Watching someone wealthy fail or perhaps pull it off against all odds is a pastime usually reserved to TV dramas.
He's also an eccentric, with a considerable ego and a well-known internet troll. He's easy to make fun of. Given how freely he distributes insults and snipish takedowns, he's someone many people want to make fun of, for better or for worse. Oh, and his communication style, which could be described as straight to the point or impressively naive, has as many fans as haters.
> Why is Twitter so important?
Because a non-negligible fraction of the world gets their news from Twitter, directly or indirectly. Also, because Twitter has become a vector for propagandists of all sides and that this propaganda, by all signs, works very well.
> Perhaps this is incredibly ignorant to say, but why not let people who want to use Twitter use it, and those that don’t, don’t use it?
I don't think anybody disagrees with that.
> I feel the loudest voices are anti Twitter. Or they hate Musk. And Elon Musk seems to rather enjoy trolling these people. > [...] > I saw a Twitter conversation between Democrat and Republican House members. Absolutely shameful how these people hurl insults at each other, try to respond with sly, “witty” insults. Maybe I’m the one being the drama queen, but it feels like Rome is starting to burn and we have no one to blame this time but ourselves.
I don't know about other people. The reason I loathe Twitter is exactly because of that. The medium itself is not the problem but the algorithms employed by the former direction of Twitter increased engagement by encouraging flamewars and access to addictive content.
Could Musk do something good about that? Anything is possible, but none of the signs are encouraging at this early stage.
So now he needs to work out what to do with a vastly reduced engineering team, and loss in confidence from some advertisers because of the fall out from his first plan, and his communication style isn’t really helping him through all this.
Whether Twitter is a good thing overall I can’t say. It has many interesting groups of people on it in both my professional life and in my hobbies outside of that, and if it goes down I think it will be the first time we’ve seen a social network fail while still large and active.
Anyone who entered the tech industry in past decade+ has never experienced layoffs at this scale before. Young techies have experienced good times and high salaries. And the transition was quick -- just two quarters ago big companies were hiring techies by the hundred, now they're laying them off by the thousands. A lot of posters here assumed that in the worst case if their current gig failed they could go work on boring software for a year or so before finding something new, but now there's a lot more competition for that plan B job. And heaven help you if you're on a visa, have a health problem, or have a family -- could you be laid off next, and if so where do you go?
Despite all of the drama and toxicity, there is still no place like it. It's the first place to turn for realtime analysis and updates for unfolding events. It's the only credible global town square where you can see your favourite sci-fi author interact with an infectious disease specialist, a machine learning expert and an independent journalist.
I have a complicated relationship with it - often leaving for months to years at a time, but usually coming back in some capacity. It's messy and reflective of the full breadth of human discourse, positive and negative.
It is a directory of important people.
And a discussion forum where minority perspectives can be identified and boosted.
Nothing else on the Internet has such a concentration of utility, yet.
Might not be a bad idea to leave some of it there in case, you know, the future might need it to bootstrap its future.
I keep hearing this, but...is it true? Doesn't this viewpoint rely on a prediction that it's going to actually crash and be unrecoverable?
If you read Reddit or ironically twitter, he is some kind of clueless buffon, yes I get he is making likely a lot of mistakes and fairly crazy things, but it requires somewhat crazy thinking to achieve somewhat crazy results. Of course he could completely crash and burn too.
This is going to be very interesting if things work out for Musk, it will be very telling to look back at all the naysayers, everyone thinks they know better from the arm chair, lets see what results Musk is able to achieve before we jump to anything.
Regardless, right now, I'm having so much fun watching that soap opera unfold!
Apparently, I scan the article first. I assume that the initial bit is "context-establishing intro", which I can probably skip. Every new paragraph that introduces some "unrelated" topic (in this case, photovoltaic panels, CO2, US midterms, ...), I will skip assuming that it's there to add depth to a point.
Basically, I look for the point, the meat so to speak, and in this case I was confused when I hit the end and hadn't found it. Maybe that was somehow the point? I will never know because I closed the tab, in spite of the topic being something that I am personally interested in (and have written about too, in the past).
Well, you can’t just say that and not drop a link to it :p
That said, this article absolutely does not seem to be an example of what I'm describing!
You can try to make a pie crust with a hammer, but a rolling pin works much better.
How can I say this? Would it not make sense to just /read/ things this short?
* There are multiple news threads at the moment
* As a result it's hard to pick just one to blog about
* Therefore, this post if an attempt to tackle all of them
That might explain why it's hard(er) to follow - it is a bit rambly, but that is partly the point.
https://en.wikipedia.org/wiki/Ego_depletion#Reproducibility_...
Is it worth trying any other Stross works if I loathed Accelerando? to the point of being physically unable to finish it, which is pretty rare for me. I guess I'd say it felt overtargeted: the constant barrage of hacker culture namedrops, weebshit, BDSM, and similar make it feel like a masturbatory mary-sue fanfic, not a real novel. It's like: all this shit is theoretically right up my alley, but the presentation is so overbearing and weirdly self-serious (or did I miss the joke?) that the cringe factor was just overwhelming and I was left with the sensation that, somehow, this thing has got to be a false-flag work designed to make everything I enjoy look idiotic. I don't think I've had such a viscerally negative reaction to scifi since Honor Harrington.
I have no problems with Stephenson or other superficially similar authors.
Charlie Stross? Sounds like a white man. (Checks.) Yup.
White liberals projecting their values onto “people of color” is maddening. Since they coined that phrase, their margins with non-whites have eroded, to the point the GOP just had their best year with minorities in decades. In Florida, where the GOP actually has a ground game targeted at Latinos (the youngest demographic in the country) they won the demographic by double digits (and not just Cubans). Now maybe the author can be forgiven for not knowing that since he appears to be British, but over the pond a significant defection of Indians from Labour has helped make an Indian Tory Prime Minister.
As a card carrying “person of color” I’ve literally never heard a non-white person (who wasn’t on TV) express concern about climate change. I’m sure they exist. But you just have to look at the polling to see that it’s not a top concern: https://news.gallup.com/poll/1675/most-important-problem.asp...
”We find that Hispanics/Latinos (69%) and African Americans (57%) are more likely to be Alarmed or Concerned about global warming than are Whites (49%). In contrast, Whites are more likely to be Doubtful or Dismissive (27%) than are Hispanics/Latinos (11%) or African Americans (12%).”
https://climatecommunication.yale.edu/publications/race-and-...
And just because people say they care about the environment in the abstract doesn’t mean they place a high priority on it. You can see this in actual voting behavior. The candidates that focus the most on the environment, like Elizabeth Warren, have less support from minorities than ones that focus more on the economy, like Biden.
Question 41C. Hispanics care a bit more about the environment than white, but if you account for the median ages (28 for Hispanics, 42 for whites) and compare with the cross tab for age, the white-Hispanic gap seems mostly if not entirely age driven.
As a "card carrying" PoC you heard it from me now. So stop it with your uncle tom herman cain bullshit. Your POV is absurdly atypical for a brown immigrant and you must really stop trying to mislead others into believing that you speak for all brown people.
And then there’s all the brown Democrats who don’t meaningfully care about climate change. Nearly all the brown people I know are democrats, and will say they think climate change is real (hell I would say that too) but that doesn’t stop them from buying an SUV.
The people who care deeply about climate change, enough to change their lifestyle, and vote on the issue, are overwhelmingly white. For example, this picture of Extinction Rebellion NYC has to be the whitest gathering in that city: https://images.squarespace-cdn.com/content/v1/5ce0b9b4c14cff... (I’m not cherry picking. That’s the first group picture on their website.)
It’s bullshit when white liberals try to appropriate the political capital of brown people to push issues that they care about more than any one else. It’s a form of political disenfranchisement.
https://en.wikipedia.org/wiki/Extinction_Rebellion#Arrest_as...
The author rails against but ONE of the major threats we face in our economic & political systems, while actively embracing the evil authoritarian beast just across the way.
I have a hard time respecting anyone who so easily sacrifices individual liberty in order to chase out the bogeymen of the moment.
Can you say what this is referring to, because I’ve read the piece and I can’t figure it out.
I'm familiar with the social conservatives who believe that what the US gov't is doing with regards to marriage amounts to an infringement on their rights. Not sure if that's the argument you are trying to make here.
What's wrong with that? He's not trying to be a journalist in some mainstream publication (both-side-ism is suspect anyway). You read Stross to get his take on things, from his point of view.