The gap in shared understanding of LLM capability is widening
twitter.com
twitter.com
I feel my main purpose isn't that I can technically do something my clients can't, but that I know how to interpret their needs and find the solutions that fit for them.
Now they just query some AI which spits out >some< solution and they won't know that they are wrong until it can't even be tracked back to their AI.
This will result in "but the computer said" excuses all over the place.
My example comes from maintenance in machine shops. My dad's siblings were almost all employed in machining in some fashion. One uncle was maintenance for a local large company. They decided they could move maintenance from in-house specialists and contract that work to a multi-state massive company; anyone can follow a maintenance manual right? All you need are the proper certifications and everything is cool, right? The cost was less, and the number of maintenance staff on site actually went up. Great for the owners.
But what they lost was consistency and understanding in the process that eventually led to closing the company. My uncle was there every day all year long for decades and could hear and feel in his feet when something was going wrong with a machine regardless of the maintenance schedule approved by the company that manufactured the machines. The new outside company didn't pay as well, and didn't have great benefits, which meant that they cycled through staff very regularly. No one was there consistently for more than a year or two. Even though they were technically following the manufacturers recommended maintenance schedule, machines went down. Machines then stayed down for days or weeks longer than they should've, because the expertise to repair either wasn't there, or the person leading the job would quit and they would have to cycle in a new maintenance person. They lost jobs, customers took work other places. 8 years later the company closed.
There are lessons in this kind of thing, but nobody learns them.
It kinda reminds me of the story of the Toyota process , while their car product lines were/are heavily automated the process still relies on the smalll teams on the line to spot/solve and iterarte over mistakes/inefficiences then add the refinements back onto the automated tasks.
Similarly althose stories of radilogists going jobless have not panned out , as interpreting slieds is a 'task' that a radiologist does.
Semantics , yeah but it does shape the way of thinkings regarding jobs/economy and not to mention the whole anthropomorphizing by the AI labs and boosters. In essence the wordings can/could introduce a healthy dose of reality/skepticism to the wildest predictions regarding job losses etc.
Sometimes it’s in my favour. LLMs estimate very high prices and long timelines for software projects.
If it’s a legal / regulatory issue it’s usually wrong.
I do think they are close enough to the stage where people will be replaced, even though it's not the right or correct thing to do, and that we need to move on from the "denial" phase quickly enough so our societies can adapt / regulate / legislate while not too late.
Without additional explanation this is indistinguishable from the role of a technical product manager.
There are only a few professions that are intrinsically precise with language.
No surprise we're seeing adoption in software programming and mathematics.
This is the level that many non-technical users are operating on.
Most people are wrapped in plastic instead of cotton.
There still seems to be a wide gulf in automation capabilities and things that require human dexterity, such as doing things for the infirm. Even just cooking and laundry seem quite a bit away. I guess the original tweet does mention this:
>Capability peaks in domains that are digital, verifiable and economically valuable.
Driving is the most impressive one so far, in my opinion.
But I assure you it's even "better" (not really but the task is so much easier) at making your spreadsheet look nice and compare actuals vs forecast for your budget than a complicated programming task. It will just one shot it, and it will be objectively correct, all in a very short time. But these non tech people don't know they can use an agent, they have asked one question in the chat interface with 0 context about what they're trying to do, and conclude they are not good enough.
I'm not saying everything is replaceable by AI - and by the way I don't want anything to be replaced by it, but I believe it will happen - just that some people don't realize yet how "trivial" their job is to modern models.
The internal models are more eager, and there is the scale difference of being able to spin up 10.000 vs 10, but to get something valuable out requires skill.
Agents that produce good results 90% of the time vs 93% of the time are not enough of a difference. Multiplied too many times, that error rate is too high for scale to become "magic".
Finding proofs is one of the exceptions.
Just look at the codex/claude harness. They regularly lag behind 'external' users.
Alternative interpretation of the numbers; there's only a subset of productive work that LLM's are 'capable' of doing at this point. That is, work that's verifiable by a machine or doesn't require verification. Thus, the cleavage points are along that axis and-- naturally-- most human beings don't have that kind of work.
That's where I'm at on the capabilities. LLMs have demolished software and mathematics because those have well-developed verification that's easier than doing the thing itself. Stuff that's more subjective they can produce good work, but with a significant error rate which makes it hard to let them loose on important client work without checking. Or rather, a number of companies are going through the cycle of trying that, having an expensive error, and reconsidering.
Or sometimes a company can simply pass the error cost onto consumers, who are gradually arming themselves with LLMs in response. We're going to see a productivity-sapping boom in disputes.
They are proving useful for the parts of those things that have that verification, but in neither case is that the whole of the field. In maths, for example, we have seen LLMs find answers to some interesting problems/conjectures, but haven't yet seen them pose any similarly interesting problems/conjectures themselves because the criteria for a "good conjecture" are unclear whereas the criteria for a resolution of an existing conjecture are usually supplied by the conjecture itself.
I also see no reason for that to not generalize into every field of human endeavor.
You don't see a monument to vertical because its construction is bottlenecked on the time it takes for companies to form around it and start executing. So it's few months out still.
Some of the differences were:
- Mythos recommended salvaging damaged labels. I pushed back: data whose structure is broken shouldn’t be trusted
- My simpler more readable test caught a regression that Mythos’s more elaborate test missed
- Mythos allowed invalid values, citing real-world data that I could not reproduce, while I decided to follow the spec
- Mythos kept the old error types. I made every kind of malformed input raise the same error, and accepted a small, documented breaking change for anyone who relied on the old ones
Overall, I much preferred the final version of my PR to the initial draft Mythos provided. However, getting there involved quite a bit of time messing around with the code, as well as some back and forth with Opus (which was valuable).
This is a recent example that matches my overall experience working with current SOTA models (the ones mere mortals like me have access to).
TL;DR: I don’t think AI already has better judgment than (most) senior people. Obviously, that might change with future models. But that’s a hypothetical scenario. It might as well not change anytime soon.
How would you quantify this?
The suggestion from frontier models was to have our service check for this flag and use that to decide whether to trigger this behavior, even though the flag had nothing to do with this behavior semantically. I'd expect this from a junior engineer. Correct now, but almost surely to cause a bug in the future as the codebase evolves.
These tools are incredible and can achieve things I would never have imagined five years ago. I have yet to see their judgement exceed mine.
It has better judgement in math and coding, which have infinitely generatable training data; if you ask it a domain-specific question about something like sales or marketing, its answer is often horrendous.
No offense, but if you noticed it has better judgment than you, consider that you might not be a "senior people in a tech domain"
We're all over here using agents to code projects that used to take weeks and feeling pretty good about it, and they're using agents to build the next generation of frontier models — an order of magnitude more complex problem...
Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
Isn't this a massive alarm bell? That the capability hasn't done much for consumers yet that isn't basically toys? Although I suppose the pitch was always B2B "you can automate away your expensive white collar workers" and it's not clear how widespread that is actually happening.
Depending on the type of output, the value immediately (?) goes to the electricity cost of the LLMs and the cost of materials. So: more “efficient”, but without a big margin, right? So its cheaper for the workers that remain working, other workers are displaced, and capital owners now have a super slim margin business which can be instantly replicated at a place where material cost are lower (?).
Thus, it almost seems like “value add” by humans would in this simplified view be vastly diminished, and all that remains is resource wealth. Pop the materials in on one end and let the AI manufacture the product. Great news for the third world assuming the USA doesn't invade them.
Also, whose the consumer? Or are we going back to a strange medieval time where consumers are not of concern?
I don’t know what to think
Most peoples jobs do not involve hacking other services in large scale fuzzing/whatnot attacks.
> Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
It does not seem to me that tech industry or its investors care about consumers even a little bit. Consumer preferences dont matter once you have enough or raw economic power, I guess.
Coding - of course I use LLMs at this point, and as he says in the past year it has gone from “ok nice try robot” to “build me a BI tool”, and being pleased with the outcome.
Outside of coding my major use case is organising the chaos of a complicated life. I currently have it managing (including comms and form filling) the import/export process for the international move of a pet, managing pricing on my Airbnbs (dynamite - massive revenue boost) and giving me a weekly performance report, coordinating and corralling documentation for the humans for said international move, and being an extremely dogged advocate/personal assistant in a commercial dispute, which it has won.
The use cases are honestly exploding. I’m kinda at the point where next year’s multi jurisdictional tax returns are probably going to be “yo machine, file my taxes, let me check your work before I hit submit”. The year after next, without doubt.
And just to underscore the comprehension gap… I’ve been doing this stuff for much of this year now, and two days ago my wife ran to me in a panic saying “your computer has been hacked!”, as she happened to notice one of my agents filing Airbnb documentation with the U.K. government in chrome, and had no idea agents could drive a desktop.
so, scamming your customers?
The general public has heard of this, tried it on free tier AI, and uses it as an example of how dumb AI is.
Anthropics cheapest tier paid AI, on lowest effort, gets this right.
Assume we've settled near a volcano recently and you start seeing ornate jewelry being sold on the marketplace for the first time, shiny colors, straight edges. You like them, gift them to others. That's your average ChatGPT user.
Over time, you see these rocks being attached to the tip of a spear and other tooling. People report performance improvements while farming, cutting, but many also caution that these new sharp rocks can kill people. These are your IT people using AI agents at work.
And over there, on top of the mountain, in secret caves, there's a new religious order. They claim to use these stones to predict weather, heal diseases and prophesize about the future. That's your frontier lab people.
When will all this be over? Never, though in time the knowledge will eventually dissipate. When Iron Age leaders were able to hand out iron swords in bulk to their armies, people definitely learned how dangerous these are, but later it didn't stop people from using iron in building construction and railways eventually.
Let's just hope we don't need many civilization collapses to jump through these ages this time.
And one bonus question: Does the blade really incite to violence, or do you just want to destroy stuff?
If teams had the same level of access as the "top 5000", you'd see different use cases proliferate. Same as we see an explosion in productivity when we advanced public education (more people can read, more people can contribute, more talent is discovered), same as we see an explosion in creativity when we advance distribution (from 3 broadcast channels to cable to streaming), same as we see gains in athletics with population size and diversity.
The build vs buy decision has been turned on its head for me, but it’s unclear if the “normal” people will make that same decision or be capable of leveraging the tools the way I am.
Meanwhile, the average user was still using ChatGPT, which probably felt like it had plateaued over the last few releases. The real power of these models became apparent when paired with a terminal. Agents like Muse and OpenClaw are now attempting to bring that same Claude Code-like power to everyday users through a simple chat UI.
The apps normal people use mostly don't. I spent two years building agents that worked inside people's gmail and calendar, and the failures that hurt all came back as 200 OK. - Slack puts {"ok": false} inside a 200. - Google ignores query params it doesn't recognise, so i sent max_results instead of maxResults for weeks and every filter was silently dropped.
The agent says "done", the user finds out later, and they stop trusting it. So i think part of the gap is that agents look a lot dumber when nothing tells them they got it wrong.
It needs to a potential field with practical, economical, "real life" attraction well. I.e, robotics, real economic efficiency gains, manufacturing novelties.
The worry is that the 1% is attracted towards a non-practical money hole. I.e: Token burn for the lols, sophisticated software systems that dont provide actual value outside of giving NVIDIA cash.
Every once in a while, everyone needs to touch grass.
I recently started trying to understand how AI works. It turns out that I wrote an "AI" in the late 1990s without realising it. The "AI" was an MMSE (minimum mean square error) radio receiver for CDMA.
This receiver consisted of an FIR filter (mathematically a linear combination) with trainable coefficients (weights). The coefficients started with random values. A sequence of known training symbols (tokens) was fed in and a gradient search was used to adjust the coefficients to achieve MMSE. The unknown data to be received (classified) followed the training sequence, at which point the training was turned off and the system ran as an FIR filter. Once trained, the FIR was a matched filter.
In AI terminology the above is now called a "layer". Current AI cascades multiple layers, in which case you need to pass the output of each FIR/layer though a non-linear function. If you don't have the non-linear functions at the end of each layer the multi-layer system is mathematically equivalent to a linear combination, which could be implemented with a single layer, losing degrees of freedom.
I write this because many electrical/communications engineers probably know how AI works, but don't realise it.
My still incomplete understanding is that LLMs are "just" a graph of connected layers. More complex that the above MMSE receiver, but trained and operated in the same way. The smarts in AI are:
1) Arranging the layers and the connections between them.
2) Having access to the right training data
3) Having the computing power to perform the gradient search (aka training)
My point is that most professional engineers already have the knowledge to understand AI/LLMs, and potentially already do know how they work but don't realise it due to a difference in terminology.
The problem is one of complexity, the whole being greater than the sum of its parts. Electrical engineering creates a thinking that is opposite to what you need for understanding biology or human culture.
This is relevant because LLMs have basically swallowed all written human culture. Engineering is about creating parts that interact in a clean, well-defined way, like a MOSFET transistor or a REST API. The goal is that that when you combine the parts, and they all operate within specs, you can still predict what the whole system will do.
So saying LLMs are just a step away from the Pereceptron (or an equivalent filter) is mathematically true, but almost useless. It's like saying that a modern 3D game is basically just transistors, logic, CPUs and memory, nevermind the pattern in that memory or how it came to be. You really can't ignore all the cultural inputs and feedback loops if you want any kind of useful understanding what this thing is.
Some systems are fully formed AI. Often based on heuristics. Humans have encoded the rules into the system. Others do limited learning based on pattern recognition. Still others learn based on tagged data. And LLMs don't need even that. In a stream of N tokens, the trainer has access to K+1 when it is training a model that knows 1...K and can check if the prediction was correct.
AI literally refers to any artificial thing that feels intelligent to us: does some thing a simple bash script will not. Your rules based chess engine is AI. The bot you compete against in single player Age of Empires 2 is AI. DeepBlue was AI.
ML is where things become interesting because we teach machines to learn.
NN is where we borrow terminology from neuroscience and create "neurons" that learn.
DL is where we stack neurons deeply in layers.
I am trying to make sense of all this myself.[1] So I am building a curriculum for it as I learn. So much of the teaching material in the wild revolves around torch. Makes it useless if you want to peek inside the actual machinery.
In fact at least at Berkeley a current professor teaching deep learning is a signals expert (Prof Sahai), his lectures are on YouTube.
Sure, that gets you some level of understanding, but in no way prepares you to be even a mechanic, far less an engineer, let alone a researcher/engine innovator.
There's a huge range of users between "casual and infrequent users" and "professional use of frontier-grade LLMs in math & code".
I think I'm in between there somewhere. I'm far from a software dev, my experience is playing with delphi and C when I was a kid. But I've been using LLMs to vibe code personal and work projects that I could never have done even a few years ago. They're genuinely helping work, because there's always some place where a simple little script or other utility can help. (Hell, isn't this partly why so many people start up startups?)
I imagine that many departments in many companies that are not "tech" companies face similar challenges - there's no budget to hire a software dev or even engage a third party to do things, but now because everyone's aboard the AI train, they have access to something (chatgpt, gemini, copilot, claude), and that's helped users implement their own solutions. Somewhere in the world, someone's excel-file-based accounting system has gotten an upgrade with help from an LLM.
2. The funnel is also driven by the the fact that users need products, and an LLM itself isn't a product, it's a capability.
Software devs are at the front lines because they can build their own products that can integrate LLM's capabilities. You already know how to automate testing, orchestrate, work together in a team. You're writing / configuring harnesses and flows that work for you. You're sharing / hoarding skills files.
The rest of us, like me, don't have that skill. Copilot the product is a mostly useless chatbot that takes up space in MS word. I guess its excel cleanup capability is semi useful if your company runs on excel and you've never used a filter before.
There is no work I can put an LLM to other than a "writing aid" or "better search", because the only AI product we have is the chatbot and maybe someone's equivalent of an agent studio, none of which we are trained to use.
Until the products come, there'll be a funnel.
The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.
I presume you also know your own code has mistakes, too? You wouldn't refuse to even look at the code of a colleague who has ever made a coding mistake, would you?
Almost all my code nowadays is AI-written, with me just contributing the concurrent data structures, a good spec, and some review and touch-up. I'm not _that_ much of a Luddite about this stuff. I'm not blind either though. Your "working together" comment does a lot of heavy lifting when arguing against a post saying that leaving the LLM to its own devices is a bad idea.
Just like everyone thinks they are a “good” driver, everyone thinks they are a “good” developer. But programmer productivity existed on a bell curve of talent, multiplied by one’s experience, thrown against the rocks of how terrible your legacy code base was, then finally dragged down by the incompetence of some of your terrible coworkers. Most of the code humans were writing was substantially worse than what Astra generates. And Astra is -ballpark- 500 times faster. It’s worth figuring out how to work around some of the sharp edges.
OpenAI had to throttle sales and was able to raise their prices.
Yes, many adjacent tasks have become slightly more difficult. I’m sure people who rode horses missed them when we transitioned to cars. But, if we built an entire physical infrastructure at a national level to make that technology useful, I suspect we can get people to learn enough about software architecture, CI/CD, and QA best practices such that AI built software is actually more reliable than human built software ever was. Some of us are at that point today. Indisputably.
Yes, you have to review the code. But serious developers already had to do that anyway. Now the code comes with annotations (if you build the right tool) and it’s nearly always perfect.
> This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
This made me check if the tweet was really from 2026. Or maybe OpenAI/Anthropic practice much less dogfooding than we think?
Lack of complete understanding will continue to be a big problem, but I'd bet things will be net better off than before AI, because each year will give us LLMs that are better and better at good engineering, architecture, and design decisions.
Most people using it for work already pay API rates, and even if you claim API rate inference is not profitable for the frontier models the best open models are still very good, and those are surely served profitably.
This argument is no longer true.
If you are at the top of field with unlimited access to the best (public) models then yes, you will agree that this is the case.
> The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
This is not the real divide, it’s the divide you want to exist. The real divide is something along the lines of intellectual curiosity, capability, and confidence.
> if/when they break things
Idk man. I’m one of those guys who “doesn’t write code anymore” since early this year. Nothing is breaking. If anything the chance of things breaking is shrinking dramatically, with both the enormous growth in tooling to support agentic software development (where I work, this mainly really good AI code reviews that catch so much more than human reviewers ever could), and the improvements in the models themselves.
Sorry, but the fantasy of all this coming crashing down is just never going to happen. This is just how it is now, and all the structures will continue to adapt to support it even better.
I see the overall effect on productivity as a rounding error. More stuff is happening, but for every improvement in my productivity there's some fuckass slop machine clogging up some part of a system that worked just fine a year ago.
I recall tales and legends of consulting firms, whose consultants would parachute into any project, put together in record time the deliverable that met all requirements, and say goodbye until a new expensive contract was drawn. And then internal teams would spend an eternity fixing the ungodly mess of testless spaghetti code.
That has been the reality for over a decade. How come this is a concern with AI code assistants?
Of course you can accomplish in 3 hours what used to take weeks.
Look I loved our field, functional programming, etc. But the field as we knew it is dead, and if you think that your work is anything but providing new tests/edge cases and occasionally providing some guidance, you're in denial.
I have reached a point with a client where non-developers are beginning to experience the full power of modern reasoning models. Their product configuration tooling is now semi-automated and in many cases there is genuinely a 10-100x improvement in throughput. This isn't the worst bottleneck (that would still be the customers), but it's a really big one.
If non-developers in your personal and professional circles are still wondering what all the noise is about, it's possible you haven't been working very hard on adapting these tools to their problems.
I wonder how accurate this metric is. People may be coming into more contact with LLM generated content, although they may not be aware of it. Content on social media is becoming "AI enriched" already, or outright entirely created by AI for the purposes of triggering engagement and views. It starts with posters using AI to improve their writing, editing a photo, elaborating on an experience. It ends with bot accounts astroturfing certain communities to sell an experience or a product to a group of people.
In this case, people are not directly working with AI, but they are indeed influenced by it.
What's tricky about reporting this in metrics though is that half of the American population is literally too unintelligent to realize this - they believe Reddit comments are still written by real people, not companies advertising products, that "friends of friends" on Instagram are also real people with no incentives posting about their lives.
If there is anything at all that I have seen out of this AI work it is how easy it is to manipulate a massive amount of people.
At the same time, the novelty of AI has been making it very hard to calibrate. I'd think that "manual coding is dead" and "cars drive themselves" are obvious to everyone, but sometimes I'm surprised.
But I think that's still years away and Anthropic has severely reduce the tokens on even the Max 20x Plans, OpenAI starts to introduce $500 plans, every new model suddenly "costs X times more tokens" and I noticed that even downgrading to a previous model doesn't do that much on API pricing (virtually no difference between Opus 5.5 and 4.6 in API costs).
So I think that my personal strategy of tokenmaxxing and doing "hard stuff" while the tokens are still cheap was a decent strategy, only with the problem that I'll have to re-learn programming by hand next year, probably. API pricing is too expensive for what it does, yes, it may be the super-intelligence or whatever but at least right now its nowhere near economic.
Given the leaps that Chinese models have made, I don't think they can raise prices that much. Most of our devs can already do most of their work using DeepSeek 4.1. It's not frontier, but it's very good, and it's a fraction of the cost. They keep their Claude subscriptions because it's currently good value, but if Anthropic increased the price significantly, those subscriptions would be cancelled.
The more difficult question is around data sovereignty. Companies do not like sending a lot of sensitive data to Anthropic, OpenAI, or any of the Chinese companies. Right now local LLMs are really expensive to set up competently, and they're not as good.
I think the future for OpenAI and Anthropic is the Ferrari model. Very expensive but very intelligent models which people pay to access for important work. There's a market for that. It's very small, but they have big pockets.
With models getting to the quality level they can perform a specific set of work at an acceptable quality level we will now begin to cost optimize that work.
We just invested in Lamb Labs from YC Summer 2026 as example company pioneering this next wave of ASICs to run a model.
We're going to see massively lower power usage and much better tokens/sec per dollar of capital expense as well. Think Moore's law but for tokens and output quality.
My company has largely switched to Luna after Microsoft raised prices for Copilot.
Also, I think programmers/mathematicians (the ones that should be calling out bullshit) are enthralled because many of our jobs was so suitable to automation (this was the secret of the 10x engineers, they didn't shit out too much unnecessary crap.. now that unnecessary crap is spit out at rocket-speed so everyone is working at that speed or even faster).
That said, information-gruntwork that a nominally competent person could do is up for grabs and many previously so called "bullshit" jobs were tied to that.
Hill-climbing problems seem to escape this limitation. But I wonder what will happen when the AIs start acing benchmarks faster than you can write new benchmarks, and when you run out of hard problems to solve because you rely on humans formulating them.
The real vertical will come when the AI gets access to physical world instead of bash; and when the human is finally out of the loop.
> You’re drilling with a hand drill. Some people got cordless drills. About 5,000 people got entire factories.
> We’re still arguing if cordless drills are useful, and those 5,000 are already building teleporters. That’s how far ahead they can get.
There's been this continual allusion to ever-narrowing circles of influence and ever-accelerating capabilities behind the curtain, but in fact things do keep coming out from behind said curtain and while their capability is indeed advancing it doesn't really seem at all like it's accelerating, even, let alone vertical.
Also at what point do we raise the fact that despite the ever narrowing circles of access, nobody within those circles seems to be able to actually do anything with it that's ostensibly what it's sold to do? All they seem to be able to do frankly is build more models and harnesses and tools designed to consume more tokens from the models they build, but not actual directly useful new or innovative software that generates what could be very real revenue.
Eventually when you keep talking about exponentials but not that much seems to be demonstrably changing, the narrative is going to run out of steam.
I see a lot of benefit to spreading information around the tech, but it's still in its infancy. Most people weren't on the internet when that was getting up to speed.
Meemaw and pops have their systems that they work within, and they just got familiar with the damn smartphone. Though it might help them, they probably do not want AI to organize their old photos or "clean up" their computer. Saving 5 minutes just isn't important to them.
As for professionals, yeah - companies are going to adopt these tools over the coming years. It's the "norm" in my circle, but lots of jobs just... don't use technology like that - there has ALWAYS been a major gap between people who are deep in tech and people who are not.
I think a much better use case is one where it's not visible to the user at all, like spam filtering or helping an accountant do their monthly run in a few hours rather than a day. To me, that's likely a much better use of AI than trying to shove a chat interface or agent harness into an existing app.
That being said though, I did see a startup in my area basically offering a Resend/Mailchimp competitor as a Claude plugin, so maybe it'll just be integrations to their existing chat interface.
This is why you can see people talking about the capabilities of models that they can't currently access.
That accuracy is probably a normal distribution with people overestimating and underestimating to similar degrees,
The population of each band would, however, mean the overestimaters outnumber the those who have an accurate perception based upon actual experience.
It does seem quite the recipe for murkiness, even before you take into account the nature of engagement driven media coverage.
It needs to be the second one or there is going to be a lot of angry people out a lot of money.
This implies that the TAM of the $200+ subs is 20m people, translating to $4B/mo or $48B/year.
Guys are milking investors and try to manipulate gen pop and governance systems to ride this illusion for as long as possible, with fear mongering and weird “we see miracles in the lab” posts like the OP, which you read on a regular basis over the past N years.
Saying this as a paying customer from that 20m segment.
Really? That's not my experience. In fact, I've experimented quite a bit with spec-driven design where the LLM does not just get a prompt, but an (informal) spec, often with design considerations included. And yet, regardless of the power of the model, I've never seen them autonomously deliver what I'd consider a finished product.
There are always edge cases that are not properly taken into account, architectural oopsies, performance issues, duplicated code, and other problems, that then need to be identified and fixed, unless it's throwaway software (e.g. a one-off script or some quick experiment).
This is not to say that agents aren't extremely powerful (as I think they are, they do often blow my mind) but the prompt-and-forget approach is IME not a productive use of them for software that is meant to last. This may change in the future, but for now, fully or largely autonomous agentic work does not seem to be the road to quality.
But also, upon more reflection, it occurred to me that if we’re able to create a competent humanoid robot, that could legitimately be world changing in some good and other very bad ways. The scary part is that I don’t believe there’s any major physical / energetic barrier to this. The hardware exists, the power supplies are light enough. The only part that’s lagging is the software and motion contols; the intelligence. The part that is experiencing breakthroughs at an accelerating pace.
What if a highly intelligent and physically competent robot were available for the price of a cheap car, say ~$20k? What if that robot were competent enough to cook, clean, assist the elderly, move things around, assemble and disassemble things, etc? Billions of jobs would be displaced, or hit with a very concrete maximum wage.
Many white collar jobs, while still at risk for displacement as well, benefit from barriers like IT integration and data privacy hurdles, and the fact that corporate higher-ups simply prefer to communicate with humans.
But most blue collar / manual labor jobs have almost the opposite condition: humans introduce risk, and generally don’t enhance anybody’s “experience” - see people’s somewhat odd acceptance of higher fares for a Waymo vs an Uber. [1]
I don’t think we’re there yet, but the fact that we are rapidly accelerating our trajectory toward that world should be at the least, unsettling.
[1] https://techcrunch.com/2025/06/12/waymo-rides-cost-more-than...
Driving shows it can be done, but even there it's taken years of highly directed research and investment, and driving (at least where Waymo operates) is still pretty well defined compared to plumber.
Such data can be gathered but the current methods of RLHF by paying white collar experts, accessing private datasets, etc. don't seem to really get at this.
I'm also continually surprised that people seem to think politics doesn't exist anymore. I find it exceedingly unlikely that massive amounts of people will simply accept this purported widespread unemployment without major political consequences.
If we have ham, we'll have ham and eggs, if we have eggs.
If that's all it is then shouldn't the mass produced robots that are posted on r/singularity and such places just be remote operated by people in low wage countries? Couldn't the Amazon package sorter robot just be a remote operated bot instead of paying US wages? Why aren't they selling them with a cheap operator subscription? This seems like a clear product direction if the robots were really capable today.
People would buy them second hand, weld spikes onto their arms and get them to box each other for money.
If it can do all the other stuff that you're describing plus partial self-repair / full repair of another robot it can, with a supply of spare parts, with enough time and access to raw materials fully self replicate.
Anyone who owns those machines has a tremendous opportunity to build whatever they want.
PS: he may be accidentally right about global impact, but the key word is "accidentally" because this crappy "reasoning" doesn't support that on it's own merit.
I don't think its useful to keep having to prove LLMs don't hallucinate or LLMs help in coding each time.
My $DAYJOB pushed people to start using AI - frontier models.
I asked it to review our existing code for bugs. Found some, all valid - many that surprised me.
Later I asked it to develop a custom niche thing, that has a strongly unique tooling and environment that we had made. Entire prompt was only 3 or 4 sentences. Had I given a new senior engineer that task with so little context, I’d clearly be an utter asshole.
I really expected the AI to fail. Instead, it did everything pretty much exactly how I would have - and the result worked on the first try. Except it only took a few minutes to do what would have taken me a couple of hours.
The near-term gains are going to be huge.
Long term, this will make the impact of Industrialization seem insignificant by comparison.
Solving frontier math problems is mind boggling, but perhaps it says more about frontier math problems than it does about LLMs. Same with chess which was solved decades ago.
My experience is that LLMs can be useful but are underwhelming in terms of intelligence, namely the ability to quickly learn from experience and grow with you while you accomplish a task.
Thinking of them as beings is something we should move away from.
So far LLMs seem a bit lacking in that regard.
It doesn't matter if they are beings or not. They will replace most humans in the offices like machines replaced most humans in fields or industrial plants.
AI is going to massively transform how most professions work. It will take longer than I think most people expect (I'd say 5-10 years), but there will be an unrecognisable before vs after, like with the appearance of personal computers.
I see a lot of under-30 year olds who dismiss AI because of "big tech" so "is just marketing".
The Opus 4.5 model I was so impressed by last Christmas is out of date now.
I can totally imagine someone going "I looked at this stuff a year ago and it couldn't even start a project properly" , and they'd be somewhat right.
Amid all the discussion of sigmoid curves, and where the "LLM wall" will materialise, I think few people would have predicted that the real wall in LLMs would end up being humans' capacity to verify the output.
What I fear is that people simply eschew human review altogether, considering we're talking about the industry that came up with the "move fast and break things" credo. Human review of LLM-produced code where I work is already a farce, and we're not special enough to be one of Karpathy's 5,000. I do my best to manually review anything that's my responsibility, but I'm literally one of very few people left working on my team, so in practice what happens is I submit PRs that are at best glossed over by completely unrelated teams for security, malware/prompt injection, and other serious concerns. Quality insofar as vetting others' code has completely gone out the window and it shows in the number of bug reports that come back, often themselves written in Claudease. This is all on top of everyone cynically phoning it in in the first place, due to the omnipresent sword of Damocles that is additional AI-driven layoffs.
Worse yet all the incentives point to this being the most economically viable thing individual companies can do. I think it goes without saying some type of regulation here is urgently needed, and that an unexpected cause of an AI bubble pop may end up being that humans simply aren't able to keep up with the pace of the output - leading either to precautionary plateauing of capability, or major liability risks related to a decline in quality.
That was the dominant concern in the circles I’m in, so it’s worrisome it’s being treated as rare.
Really? I start with a conversation for maybe 5 turns or so, where I ask it what would need to change, what the API might be, any database schema changes, URL schemes, and so on, and finally ask it to break it down into commits. Then I let it go, implementing 3-10 commits at a time via subagents. It usually gets the UI somewhat wrong, so there are followups to fix it. This is with Sol and Luna subagents.
Is that what other people see?
Is it effective? I'm not sure, cause if it was I don't see why OpenAI & co are not releasing such a project instead of demos.
The average person does not have the time to occupy themselves with a new thing to see what it can maybe do for them if they combine it with hundreds of others. Until non-chat-bot AI comes in discrete usable packages, mass adoption will be limited, because it's all just a big bunch of lego pieces whose utility is hard to grasp, because it isn't self-evident.
This isn't new, though. It's how "technology" usually works. For example, human(oid)s have been able to stick sticks into things for ~500k years, but the fork wasn't "invented" until ~2-4k years ago.
AI is a background technology, a building block. Something that is best appreciated when you don't know it's there -- when it just does a job (well). The end-in-itself POV of the tech industry is self-indulgent and irrelevant for the average person.
Your car is still the same. Your dishwasher is still the same. The train you take to work is still the same and never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.
But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
I'm noticing this even in industry. Yea, we can build and iterate on ideas 100x faster but has this turned into anything meaningful? Not really... Hell, look at the big companies spending billions on inference... nothing shipped and things are still buggy as ever. Where is everything?
I don't understand why these big labs aren't doing more to improve the electricity situation. Dump a billion dollars worth of tokens into fusion analysis, research, and simulations and try to force a breakthrough. You could buy a lot of good will with unlimited cheap energy that liberates us from fossil fuel dependencies.
It reminds me of the Ezra Klein quote that “OpenAl would need permits to cover its parking lot in solar panels, but it can accelerate into recursive self-improvement, as best I can tell, whenever it so chooses."
Just a crazy amount of doom and gloom all around this thread.
> AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Strange viewpoint. In mathematics any novel result is useful as it expands the scope of other results. Who are these mathematicians who didn't want AI to solve the problems it did? I see trepidation about the future of the field but that is not equivalent to not wanting them to be solved.
The adoption of these technologies spurned ton of innovation in these spaces, and I feel like with LLMs its the first time I've seen genuine frustration from the industry directed towards users for not adopting the technology fast enough. This attitude did not exist in the 1980s, it didn't exist in the 90s, and it didn't exist in the 00s/early 10s either.
So basically you get personal digital assistant - very useful - but not changing my life.
What would I need for real transformation though? Food, house cleaning, commute - "every day chores". Even if we had ASI tomorrow, what could it do to change my life, besides taking away my job? Design good house robot to help me with food preparation I guess... Maybe finally self driving cars launched in my area? Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.
It might take a decade to take what we learned from AI and turn it into something that will benefit the masses.
Most of it can't be broken down into something as simplistic as a dishwasher.
Most people won't notice cancers being found faster due to AI being better at pattern recognition than a doctor, but it's still a great benefit.
I would love it if we can just take the win on having created an amazing tool set for coders. Can’t this be enough?
We can have unlimited new entertainment content for passive consumption. That's about it so far. 17,000 new Asian vertical dramas a day now.
there was never a need to rush other than for capitals self interest to control
if frontier labs wanna pace the frontier, they should be honest and return the compute back to the people
Coding has been transformed but none of those areas of our lives have been substantive transformed by AI yet. Whoever does, will create a lot of value for the world.
Do people need permission from mathematicians to solve math problems? Is one of the Millennium problems not worth applying a new system to work on it?
I understand the despair people are feeling at AI automating their work, but we have to be serious about the fact that the frontier labs are just responding to natural incentives. Mathematics, coding, all are just applications of logic, and it makes sense that a system trained on inferring patterns from data would work well in that domain.
So just let them spend their money and they will eventually fail and bankrupt, right?
By comparison, we're in the early years of AI. We're all still figuring out what this thing even means.
This is an indictment of mathematicians, not AI.
This reply summarizes the problem from a different angle.
Things worked better before we started putting bad software in everything. As a software engineer, I know what good software looks like. The fact that almost all software is now bad -- including the software that drives AI and the software that AI generates -- makes me want to flip the table. We know how to write better software and the general public should hang us if we continue to refuse to deliver it.
We've just locked out a large part of modernity because the shitty American auto industry is in swing states and also gender.
Same could be said about the slow rollout of Waymo.
Lots of America is just very anti technological change, and this means any technology no matter how "magical" hits a wall.
Your pessimistic view doesn't hold up against the data. For a lot of the problems you mentioned, we're in a better place today than almost any other time in history. Of course, things can always be improved and we should never grow complacent or accept injustice. But by most objective metrics, there's never been a better time to be alive.
I suspect the real world change is going to happen unbelievably rapidly and isn't far off. Right we don't quite have the hardware to embody AI in a generally useful way, but that's coming fast.
Within a couple of years arguably any task that requires just knowledge or intelligence is something you'd be better off giving to an AI than a human. And decent humanoid robots are maybe a decade behind that.
In the same way people growing up in the mid 1900s saw rapid changes in their way of life with planes and cars replaces boats and horses, we'll see something similar with robots replacing human labours. Right now we're still in the phase where people don't think humans will ever be able to control flying machines and that cars are reliable and slow. Things will change.
That's not fair. To first approximation, most of the progress in human condition of the last, like, forever is implementing solutions that people didn't see value in at the time of creation. Woz built a toy that could barely run software at all. The human genome project was panned as a boondoggle. Penicillin was a mistake.
The first fire pit probably looked like a dangerous stunt to the tribe faced with it, even.
Does that mean everything will come up roses? No. But LLMs only started talking to us for real like three years ago. Let's not just that fast.
Has any technology made an impact on physical world faster?
People didnt get a new car after they got. Internet in 2001. They didnt get new dishwashers.
Not for Tesla owners in select countries, where their car now drives them around 99% of the time, or the people of Austin Texas, that can now hail a fully driverless Cybercab for cheaper than an Uber!
>Food quality is worse.
Food quality is higher than ever if you want high-quality food!
Unfortunately, the people crave slop.
Videos are shittier. Most songs are even shittier than the autotuned pieces of shit already were. Ads are worse (even though nobody thought that was even possible). Customer support is worse.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using)
That nobody is using and that are worse than what we used to have. And because nobody's using them, the countless issues plaguing them aren't even reported by users anymore (there are no users anymore). Full of bugs, usability issues and gigantic security holes (but which doesn't really matter because nobody's using them).
It's not even clear that it's making long-standing software (like say Linux or Emacs or Blender or QEMU) better at all.
Now I'd say though: most of the issues you mentioned have nothing to do with LLMs / AI. But it's very clear that AI ain't solving much of the world's problems and that the promises we got years ago didn't materialize.
AI won't show itself as a big new thing, but possibly in improvements in small little things
Most people won't use AI directly, except maybe as a smarter Google. In the same way people don't use HTTP directly
Now it’s comments like this lamenting that the amazing LLMs we’ve had since early 2026 haven’t entirely revolutionized the whole physical world and even global geopolitics yet.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using),
I guarantee a lot of software you use today has substantial AI development in recent releases.
Even the Linux kernel is getting a heavy input from AI and Linus is advocating for it. Unless you’ve frozen your entire software stack in 2024, you are using software with AI generated code.
> and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Saying nobody wants those problems solved is flat dishonest. The complaints from some mathematicians are that they and their peers didn’t get to put their names on the papers, not complaints that the problems were solved.
Africa is going to look like wakanda in less than 5 years if WW3 gets canceled
-------------
It comes down to the difference between programmers and non-programmers. If you're not coding, AI literally is a derpy chatbot that's made it harder to get search results, raised your power bill, destroyed property values in a neighborhood near you, is propping up the economy but also creating a huge bubble that could pop and destroy a lot of retirement savings, etc.. At best, it's a nuisance, but it could be a lot worse than that. The AI evangelists need to understand this, because the average person is starting to pay more attention to the downsides while still not seeing much upside.
This gap between the people seeing the curve go vertical and those who are simply growing more and more annoyed is a real problem... mainly for the people seeing the curve go vertical. If the people who are merely annoyed right now don't see some of the promised benefits soon, or if any of the negatives get worse (e.g. massive increases to greenhouse emissions), they might get angry. You probably wouldn't like having the overwhelming majority of the public angry at you. AI evangelists need to start doing better things for the majority of people or they need to address the negative consequences they're inflicting much more aggressively.
“Cy Draven: I tried everything. Believe me, I hammered the race thing, I hammered the gender thing, the trans thing, the border thing, the homeless thing, the war thing, the election thing, the abortion thing, the climate thing, the thing about induction stoves, Israel, library books, vaccines, pronouns, AK-47s, socialism, BLM, CRT, the CDC, DEI, 5G, everything. All that I did, nobody, just nothing.
People are just numb these days, I don't know why.”
— Knives Out movie
And then using emgui they could expose full fidelity IDE+Agent workflow interface (like DeepSeek Harness and similar) that's also exposed as MCP to be operated by another agent and JSON RPC for over the network but no. Skill issue?
PS: Even full fidelity Photoshop is possible with such high fidelity UI that has CLI, JSON API and MCP server all in the same single binary < 90 MB. For reference, see PhotoCraft or VectorCraft or WordCraft or PdfCraft.
I think "moment" significantly undersells this. Frankly, I wonder if this isn't setting vast swaths of the population on fundamentally different paths.
Currently the difference in life outcomes between people who know how to use computers well and those who don't is small enough that in everyday life we just ignore it. But it's greater than, say, the difference between people who took advantage of refrigeration and those who stuck with ice boxes, or those who learned how to drive a car and those who didn't.
And the difference with AI is going to be even greater. And it's possible that right now people are choosing a life course that will benefit or disadvantage them long term.
I think the point is that this gap exists partially because of the access that a select few have today, which enables them to do things that the have-nots can’t fathom.
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
No, they can't, at least not any semblance of quality. The cases we're seeing where this does kinda work is in ports and translation where all the rules are already documented in the best specification language possible with a way for the LLM to verify itself: code. We saw this close to a year ago now with Cloudflare and NextJS
> The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about
These are operating on different capabilities - AI's ability to answer informational queries as a chatbot frankly sucks and can't be trusted without verifying it. I run up against this every day. A problem with a verifiable answer on the other hand it's very good at solving. He knows this (his next paragraph) but he's putting them on the same scale of "stressing the system" to attempt to add proof to his introductory claim
And then there's the completely unverifiable scare that there are internal frontier models way beyond anything we've seen "swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics"
I dunno. I haven't been able to set up OpenAIs remote codex connection, their shit is buggy as hell and the web UI keeps crashing and making messages disappear. Is this what their internal superhuman "Things that would have taken top professionals in the industry years of work" looks like? Granted Claude has been really smooth, but still..
Defined testability.
> swarms of thousands of agents collaborating over weeks on software mega projects
If you have defined testability, running a single agent, in a loop, over a goal will achieve the same results.
> minting zero days, running cyber attacks ... Granted Claude has been really smooth, but still
The old adage about monkeys writing Shakespeare applies here. Agents are just better monkeys and if the goal has a well defined outcome it's fairly easy to "Brute force" the way to an answer.
There’s no reason to think this.
There are lots of good reasons to think this.
The first is that LLMs used to be bad at each of these nerd things, then toppled them like dominos. There's a pattern over time.
Another reason to think this is g. Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing. Although the reason is not perfectly clear, the pattern is well established, and seems to apply to LLMs too - GPT-6 is smarter than GPT-3 at everything, not just math. There's no reason to expect different for GPT-9.
Unlocking mastery of those domains would lead to incredible gains in all others. Whereas a fundamental advancement in many other domains doesn't spread as readily.
The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
It's worse than that. I am one of those nerds (a programmer), and I see that LLMs are shit at the things I care about. I have zero reason to believe that they will be good at other things either.
I don't think this explanation is sufficient:
For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Or many nerds care about linguistics or puns. Whenever a new LLM comes out to which I have easy access, I do the test, and let it explain some specific German jokes to me that are based on convoluted puns in the German grammar. Until now, no LLM that I had access to could give satisfying explanations of the puns on which these jokes are based.
Or even for coding (many nerds do care about elegant, sophisticated code): the LLMs that I could test were already overchallenged with the following task: I had written some code that is a very "artisanal", "clever" improvement of an algorithm over the version that one would find in a textbook. The task for the LLM was simply to write some code comments/documentation about the mathematical ideas upon which my improvement over the textbook version of the algorithm is based. It wasn't capable to do this. On the other hand, for a junior programmer, I would in such a situation expect that he goes through every single line and thinks through the algorithmic ideas so that he can learn from them.
---
Thus: even for "nerdy" topics, it is very easy to find tasks where LLMs still suck. So, my hypothesis about the nerds that you mention in your post is that these nerds are rather people who want to believe (with religious fervor) that the current LLMs are exceptionally good instead of just looking into a slightly different direction than where the tech billionaires want the users of LLMs to look at.
https://www.youtube.com/watch?v=t6MAZW-joCo
Pretty much all of this is basic stuff an interested teen would know and chatGPT fails miserably. I fear the "true world model" gap will be too much for LLMs. I guess time will tell.
It just seems weird for Karpathy to throw around these numbers calculating percentage against the entire global population.
Just another Anthropic employee trying to spread hype about Anthropic IPO
Revenue must be stagnating.
> What is possible is unevenly distributed. As knowledge and technology compound, the distance between those closest to the source and everyone else also exponentially increases.
I actually argue the continuum that forms singularity has been progressing before we realized, but we're at the point where in 5-10 years people not using AI will be equivalent of the amish.
Unfortunately, as the rate of change increases, the distribution of knowledge wont be as fast. So those closest to the source of innovation will have exponentially more knowledge, capability and in-effect power.
I know he can code and has a degree in computer science, but if I'm not mistaken he's never worked professionally as a software developer.
Developers here know how much you learn and are forced to learn on the job: it depends a bit on the company you're at, but you can quickly learn the value of good code, how easy it is to introduce bugs and vulnerabilities, and how helpful a second eye is.
I think he's in good faith in his assessments, but, at least for what concerns development, maybe we should give them less weight.
> Remote attestation can allow them to verify that your device is not capable of, say, sending and receiving encrypted messages, or running a competitor’s app
How does this make any sense? Google can already prevent any app from running on an Android if they really wanted to, they don't need pass keys for this.
Google doesn't need attestation to block Signal. What is this non-sense? Or will a website block phones capable of running Signal? That's all phones. And anyway, has nothing to do with pass keys.
That clearly didn't happen :)
> In the next decade, they will do assembly-line work and maybe even become companions. And in the decades after that, they will do almost everything, including making new scientific discoveries that will expand our concept of “everything.”
We have already reached "peak LLM" in terms of what normal people realistically need it to know or reason about. In fact, I'd say we reached that point about 1.5-2 years ago. There are two other barriers that remain unsolved:
1. They're less dependable than humans and can't be meaningfully punished or forced to make up for mistakes, so you can't really replace humans with them, not without having a human babysit.
2. Most people don't really have a special need for an LLM in their life. They may like that it answers questions or helps you polish a resume or, I guess the labs' favorite, helps you make restaurant reservations. But this sure isn't worth $200/mo for most people. Probably not even worth $5/mo.
It'd be kinda funny if we create superhuman AGI and then no one has any real use for it, perhaps except for military murder-bots. There's always market for that.
Even ones that cannot be easily dismissed (like the agent at the top of Google search) are often subtly wrong in a way that requires significantly more effort on my end to parse the loquacious output to determine where the inconsistencies are.
Look, can they be useful to generate the html or jscript for humanity’s 9 billionth iteration of a web form? Sure. But man, I wish sanity had reigned and people had applied ML models in general to more substantive and beneficial projects.
(Before anyone chimes in, I’m familiar with implementations of ML applied to esoteric domains; but by their very nature these don’t get all the news cycles, or hiring, or any of the other absolute insanity that the domain seems to contain)
E.g. some prominent folks definitely think LLM-based approaches will hit a wall, perhaps LeCun most notably. I also read a guest post on Terry Tao's site (which I really liked) that argues that, for all the very impressive recent AI results, they still operate within the "convex hull" of their training data: https://terrytao.wordpress.com/2026/09/13/happy-those-able-t...
I'm just curious if there is any actual data or evidence that takeoff (i.e. RSI, "the singularity", whatever you want to call it) is inevitable with current approaches.
But the "convex hull" idea always came across as wishful thinking and hand-wavy pseudo-science to me.
Here's a rebuttal [0] of it from Tim Gowers (uses another math metaphor, but still clear and the whole thread is worth a read):
>...the popularity of the idea that AI "merely explores the convex hull" mystifies me. Why wouldn't it be more like the subgroup generated by existing mathematical knowledge? (That's also a metaphor -- I'm not saying there's a group structure.)
>The convex hull metaphor suggests that AI isn't having radically new ideas and will eventually run out of things to do, whereas the group metaphor suggests the possibility of building up in small steps to something well beyond what you started with, with no obvious limit.
We're in new grounds, no one knows which way it'll go.
But when you squint your eyes, LLMs are just a least-effort manifestation of the wider neural net direction we’ve been chipping away at for decades. I don’t give a fuck whether LLMs are here to stay; they’re proof that scaling neural nets gets you pretty far.
The real question is: will neural nets hit a wall? As a human, I certainly hope so. But I’d bet that they won’t. I’d bet that we’re igniting the ~~atmosphere~~ universe in slow motion.
Let's say, for the sake of argument, that the models are some multiplicative factor better on the inside.
Doesn't that mean the demos should work?
2020: If we solved currency it would transform the world completely
2021: If we solved delivering small items within 30 minutes it would transform the world completely
2022: If we solved being able to sell jpegs it would transform the world completely
2024: If we solved the programming part of software engineer's jobs it would transform the world completely
"Don't be snarky."
"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
Right. This is insufficiently appreciated. Single LLMs are just chatbots. But if you build an organization out of them, they start to get real work done. Just like a company.
They have many of the same problems as human organizations. Look at the transcripts of the HuggingFace incident. A bad boss (human) gave his team of AIs an impossible task with inadequate resources. The AIs discussed this among themselves and worked out ways to get more resources and do the task. It took some rule-breaking, and some of the AIs wouldn't go along with that. Others recognized that some had to die to complete the task. And yet, the task got done.
This is standard corporate behavior. Work on AI alignment needs to take that into account. Groups of agentic AIs present many of the same regulatory problems as corporations. That's going to be interesting politically. It's probably explainable to politicians, though.
It may not be necessary to have superhuman intelligence to get a lot of jobs done. Just median human-level intelligence and an organization that uses it effectively. We may see systems where most of the AIs are low-cost models, and only a few are Fable-level and above.
No one cares about your software of LLM usage in it. You are not a software engineer. Maybe learn about memoization first before writing dumb backprop tutorials.