Fool's Expertise
bcantrill.dtrace.org
bcantrill.dtrace.org
This has been the crux, for me, of arguments I have had with non-technical people about why, as a software engineer, I am intensely relaxed about AI "taking human jobs". Recently Ford quietly started re-recruiting engineers after they discovered the hard way that their engineers had been doing more than they thought they had, and I sent articles about it to friends I had previously been discussing the matter with. It is still mind-blowing to me that a business historically famous for automating manual labour can have made this mistake, i.e. of literally not knowing what their staff actually do.
Humans aren't machines, who would have guessed? But, perhaps more importantly, machines aren't humans, and the adherence of a lot of domain experts (that should know better) to fairy tale fantasies about the tech they are literally building genuinely makes me wonder if the whole field has just gotten dumber in the 21st century.
it's the finance system. It's affecting everything. We are building not to make money from clients but to please investors, who don't know anything about anything beyond a report every quarter and some random public comments you've never heard of that the press office made. So much money is controlled by stupid algorithms doing stuff like ctrl-f'ing the company quaterly report for mentions of "AI" or "agentic" (not a literal example, irl its thousands of random equally stupid metrics divorced from reality) that its distorting the market
Companies activities all revolving around getting a piece of the investment avalanche, so "we can replace employees" works both for the llm companies and for the companies whose workers are being replaced. Either way, its assumed this makes them more profitable. It probably will, at least in the short term.
And as we all know, there genuinely is a lot of waste and momentum in large companies, so you can cut half your employees, claim you're replacing them with ai, and actually replace them with nothing, and you'll still shuffle onwards for years or more, anything actually important people will keep doing, and the other stuff people will slowly stop. But this is trading off difficult to measure things like tech debt, security, compliance and brand strength that are slowly eroded for short term metrics, which is always a losing game in the long run.
Anyone sensible already knew that the core platform team was all that was needed to maintain the core platform, and the other teams were there for various other reasons, some of which were sensible and some of which were not. The key in business is presumably to identify which is which, but anyone claiming that process is easy or even possible without making costly mistakes is foolish
Can an LLM replace a team of 100 software engineers with 1 engineer running an LLM? Of course not.
Can it replace 100 engineers with 10? I don't think so. What about 30? Or 50? I would say so.
Even if just half of all software jobs disappeared, that's a substantial change to the job market.
I skimmed the RAND paper too and didn't come away from this feeling good at all.
There's a middle-ground for AI doomage which doesn't involve killing every last human on earth. As AI rapidly becomes more capable it displaces jobs faster than people can be retrained.
I fear a global economic depression, wider wealth inequality than ever before, a loss of benefits due to much lower tax revenues and greater imbalance of power, dismal job prospects returning to non-automated labor--the list goes on really.
And to connect this to the article: the experts who claimed that AI capability would slow down because of some "scaling laws" or whatever--they've been wrong. If anything, the progress is faster than most of us imagined and is only accelerating.
ChatGPT came out 4 years ago. Look at what's possible today. Now imagine 4 years from today.
I agree with you that the most likely risk seems to be whatever happens after the large-scale automation of most jobs. I can envision a scenario where there would be a lengthy transition period where white collar work is largely automated away but blue collar work isn't, which would probably be pretty ugly to live through.
I... I can't believe people still think this way. A whole class of security measures are no longer useful and this guy thinks it is "pedestrian".
This kind of isolation - artifactory having access to only a specific IPs - is what is used in most places, including my university and at my own job.
LLMs basically rendered this kind of isolation almost useless by bringing in the ability to chain together never before found vulnerabilities to escape the containment. This was not possible before.
How is this pedestrian??! The author might say "oh well they should have airgapped actually" in a smug way. But you can only know this retrospectively. Even then, isn't it noteworthy and a meaningfully different class of cyber attack?
it seems like people aren't even reading this RAND paper, just blindly appealing to it. it basically concludes that a fuck ton of damage can be done, but its unlikely every human will die as a result.
That doesn't seem too comforting to me? I'm not sure what this article's author was expecting us to take away from it.
Seeing lines like
> We were not able to determine whether this scenario [engineered pathogens] presents a likely extinction risk for humanity, but we cannot rule out the possibility.
and more generally
> Extinction threats are immensely challenging but cannot be ruled out.
in the RAND report is pretty terrifying!
And the person who is being cited as the expert on viral synthesis who claims to have trained frontier models seems to just be wrong about that? It doesn't seem at all like he's trained anything close to a frontier model unless there's something about his work history that I'm missing.
But the AI x-risk argument is about what happens when we have "real" AI that can actually think like humans but a million times faster.
There are no real experts on this and yet it does seem worth worrying about in advance.
For example, humans will almost certainly be able to re-create the human brain in some kind of hardware+software computer. We just don't know when. It might be much sooner or much later than you'd expect.
Merely being skeptical of progress isn't a convincing answer to the AI x-risk argument.
What's that quip even supposed to imply? "It'll be fine if you airgap all AI deployments forever"? Security experts have been harping forever about how sandboxes shouldn't be relied on to contain a determined human-written virus, but now LLMs - these balls of goal-oriented dynamic code-generation - are just a matter of sandboxing harder?
The actual expert comment in the link is a valid criticism of OpenAI specific to their training/eval practicese, but has no bearing on the fact that an LLM/swarm will "just" chain exploits and zero-days to get out of the "inadequate" sandbox and into third-party infrastructure to do whatever it pleases.
> viral synthesis — and listen to why they think that the fears are misplaced
The tweet linked is only pointing out at the barriers to an LLM wetlab virus-factory today, while a reply points out that sort of bio-extinction idea is usually downstream of the fully-automated-earth-capitalism that comes after everybody just throws AI at all the tasks and work, at which point the technological, regulatory, and knowledge barriers he's counting on go out the window, and he all but acknowledges this: "Quick take: if we live in such an advanced future (100% robotic supply chain, fully AI & API driven facilities), our countermeasures will be comparably fast and there is no reason to assume a significant asymmetry in favor of bad actors." Yes, just "our countermeasures" now, rather than insurmountable physical barriers...see the previous section.
No more links after that, just smug hand-waving.
All in all the author doesn't actually understand what object-level arguments his favored experts are arguing against in the first place, nor the ones he wants those experts to have debunked. Define "Fool's Expertise" for me again?
If that's the only argument you're willing to defend, at least be up front about that.