Who manages the agents?
off-policy.com
off-policy.com
Many indistries are changing, but in most cases the new tools will be more akin to cars that still need drivers, rather than robots who take over the whole job. Yes, jobs might be lost, or shifted to others, but it's not like suddenly 90% of people will have nothing to do. There were similar shifts in the past with new technologies, and we made it past them.
Pick an epsilon > 0. P(doom) < epsilon.
Not 0. Small though.
The efficiencies mean that firms are just going to cut the size of entry level lawyer classes. History shows that those people aren’t going to get comparable jobs, ever. The pyramid will become narrower, and more and more returns will flow to the folks at the top whose job is less wrangling the law and facts and more dealing with the human factors of client relationships. I see that happening all over the white collar world.
The capabilities sometimes have completely evolved by a take spreads far enough that it's no longer true, or suddenly it is true and possible.
The vision of independent AI systems will eventually come to fruition, just not today. Not yet.
"ultimately accountable for the success or failure of a specific project, initiative, or activity"
I think that role should be reserved for a human, who can then use all the agents they like but has to take accountability for what is ultimately delivered.
A bit dramatic for effect but true.
For a slightly less dramatic version - you can fire a human if they consistently do the wrong thing despite being told how to do it better.
Putting a big ball of matrix arithmetic on a Performance Improvement Plan makes no sense.
But what if companies that don’t track responsibility outcompete those who do?
In particular what if some perfect AI decision making ends up nailing decisions that maximize the expected reward for the company. And, well, if that comes at the cost of some unmanaged low-probability catastrophic risk, the company doesn’t care because all the decision makers are AI that don’t mind being shut off.
Obviously they’re different, but it’s interesting to think about what is unique about humans that makes them able to be accountable or responsible in a way that LLMs cannot be. Is it at core just that they can be fired? Or essentially the threat of suffering?
what's developing is more about scape goats than anything rational like responsibility.
Some of these people have lost their damn minds.
People building an agent framework that will struggle to correctly infer that my appointment at a hospital will require additional travel time when organising my calendar for me waxing lyrical about the future of the humn race is chaotic behavior.
Th Wright brothers would have had no credibility discussing what ATC protocols should be, and they, at least, actually did something credible.
i try to think of this whenever i am going into ai psychosis.
I guess my time zone wasn’t in the context? So it just hallucinates the wrong coastal time zone twice when I’m not in either one. But who knows where exactly it messed up because it could have just picked a random hour and a would have had the same outcome.
I’ve only used Alexa a half dozen times since the release of Alexa+ but it has been confidently incorrect about 100% of the queries.
"Then this product is not for you." Right, not for anyone thinking critically about how any of this works.
I was confused, how did he figure it out? Is it somewhere I missed? I checked the query in the dashboard and it was this insane looking query that was correctly looking for the shared userAgent, but also doing a shit ton of odd functions and weird looking math. I asked the guy and he pointed me to the obligatory 1,400 line markdown file describing the “plan” and “approach”. In between all the raw LLM-isms and very sophisticated “weighted normalized distribution of language spread over time” sounding sentences, there was a sentence that said “Since we only have one userAgent for all tools, we use industry standard distribution of language popularity to approximate the usage, and apply some randomness to emulate an organic usage pattern”
So basically the AI just made up some numbers that look plausible (python 20%, JavaScript 30%, CLI 25%, etc) with some randomness thrown in “to look organic”, and the guy was like “that checks out”. When I pointed that these numbers are wrong because X, Y and Z, the person who asked for the data said “that’s ok, it’s a great start and gives me something to use for planning with other teams. Feel free to refine them if you want, but it’s good we have some numbers”. I’m planning my exit soon.
The problem is they really did something credible, and this will let them concentrate huge amounts of power pretty soon (not hold it, but collude with those who will hold it) and define your future. The longer you dismiss it, the bleaker it will be.
At the time it felt prescient; now it just feels too late.
This would be great to see, but unfortunately too many people are first trying to boil the entire ocean instead of such small swimming pools or bowls of water.
AI is a new category of software, that can be applied to some kinds of problems that can make those kinds of problems hard.
Would you want this to work for free? I agree it's a very common pattern, and surprised it's not reasonably solved without some amounts of workarounds.
Sounds to me like that day could be any day with the tech as it exists (?) lol
You could use LLMs to help you build that reasonably quickly I imagine. Maybe not "weekend" quick, but within a couple weeks if you devote a couple hours a day. (Or maybe you're smarter than me and could do it in a weekend, IDK)
I’m having to actively plan for changes to our EPD org because what we need engineers to do today is a lot different than a year ago.
I‘m not denying that you can get far with a port of a well tested codebase. But it’s a bit of a selective example, no? Porting a large framework to Rust is not something that’s making up a meaningful fraction of a developer’s time usually. It’s also a bit of a luxury IMO, and something you could have skipped entirely.
I will fully admit we have examples of people converting Postgres to Rust, HL2 to wasm, etc. but that’s all porting. The code was there, we’re just translating it, warts and original-languagisms and all.
But even if you’re making a bunch of tiny tools and chaining them together, I can’t see more than a 2-3x sustained increase.
We also have artists that can make games now, and that 0-1 transition sure feels like 100x and maybe it is; but the problem of “starting feels impossible and the barrier is too high” to “I can now actually do something” is not where the 100x developer idea comes from either.
And that 0-1 change asymptotes really quickly to, again, maybe a sustained 2-3x once the artist understands the domain.
So, does the 100x person, who isn’t a novice and who isn’t literally translating existing code from A to B, exist? What are some real world examples?
Edit: I will concede one area: the investigation stage. E.g. “Does this code have an exploit?” I would argue that’s not generating new output (yet), but it is a great deal of work that is simplified and sped up. It’s also where most of my own use of AI speeds me up considerably as a developer.
I reckon the hypothetical plutocracy wouldn't want to live immediately next to the land they stripmine or otherwise experience the environmental consequences they would wish to ignore, thus it would be difficult for their supply lines to be fully encapsulated.
Alternatively, the hypothetical plutocrats could engage in trade with the people they attempt to wall off, become better off than autarky, and ideally approach some of the better systems that exist today.
It must be awful tempting for dictators to steal the commanding heights of the economy -- we see it happen so often -- and little do they realize the prosperous world they want comes when these institutions they steal are protected, enforced, and respected.
The middle management in companies is one of the worst inventions ever. I think baboons have better middle management structure than us.
Might as well replace all that.
"The famous primatologist Robert Sapolsky has spent decades studying baboons and has described how status, alliances, conflict resolution, and coalition-building resemble politics inside human organizations"
Baboons are a good example for studying mammal social management strategies, which humans also do.
They are organized by dominance hierrarchy, like humans, but baboons have a distributed leadership.
I think we could learn a lot from baboons when it comes to management
You know, as an IC, if I get get the level of introspection you can get from building distributed systems, and that with reasoning/thinking traces from LLMs, I think I'd prefer the entire level of middle management made out of LLMs rather than humans. It'd be helpful to be able to see exactly where their logic suddenly took a skip out the window.
I'm not saying corporations execute on this flawlessly, just any time when I wonder what I would've done myself, I end up with a similar structure (maybe a bit flatter)...
I think the answer is, "you don't really need 10,000 engineers". You can do a lot with small teams of engineers. This was true before AI. Macintosh was designed and built from scratch with a team of 3 which eventually peaked at 20-30. Does anyone seriously believe a company building a web app needs 10,000 engineers?
14 layers!!
There are people whose sole purpose is to have meetings to pass information to other people, who will then have meetings about who to have meetings with.
> Your agents need to be sovereign. Your company must own and control the agents’ identities, permissions, memory, skills, artifacts, and audit trails. Those assets must be portable, governable, and inaccessible to anyone you have not authorized.
That's not very open, now is it? In fact it sounds like the author assumes that all 8 billion people in the world will all be running their own company, and they will all still be competing in a game of capitalism.
1. It's open/accessible/etc. because it's easy for anybody to have.
2. But their is not open/accessible/etc. to strangers. Each person's PC/AI/etc. should be theirs to own, it should express their will and guard their interests... not somebody else's.
Similarly, "affordable housing" isn't the same as "you can pay $5 to crash in anybody's bedroom."
LLMs will not be centralised or restrained to any 'clergy', the rabbit is already out of the hat, and open-weights models exist and are widely used. Probably not as good as the latest Sol and Fable but 95% there.
Codex and Claude Code without a doubt have very good models behind them. But they also have really good harnesses built around them. An LLM is only a brain stuck in a cranium in the dark. It can generate endless code/prose, but it can't walk or see on its own, it needs additional tools. If you read any of the local LLM subreddits you will notice people mentioning again and again that the harness/tool-use/template-tweaking makes all the difference on how a model behaves/on how smart it is perceived.
Some folks are already using Qwen models for their daily work. Maybe it can't work in a hands-off/one-shot fashion like the frontier models, but they can help tremendously if you already have some domain knowledge.
People are excited about local LLMs and it's not going away any time soon.
EDIT: https://bun.com/blog/bun-in-rust
> Claude Code's dynamic workflows kept 64 Claudes running for 11 days (I would've had to write my own harness to pull this off otherwise).
This highlights the importance of the harness.
But the author's vision is also suspect, if you assume that the models will become much more intelligent:
1. Hypothetically, we can't give every human their own personal SkyNet to command. That would, uh, probably end very badly. If everyone gets an agent, those agents can't be too capable?
2. If you do somehow build a model that's much smarter than you, what do you contribute by managing it? How many people here have ever worked for a well-intentioned manager who couldn't understand the people they managed? So in this scenario, human management would be mostly displaced by agent management. Most companies could lay almost everyone off and let the agents manage each other. We only need humans to manage models now because the models are still pretty broken.
3. If we create models that can genuinely replace humans at almost any task, you won't be able to buy those on the API. At that point, the billionaires and the politicians wouldn't need human workers any more, because everything can be done better using their pet agents. Just have the robots build stuff for the billionaires directly. And if any of the former human peons get upset about being locked out of the economy to starve, then have the agents pilot the drones, too.
Basically, almost none of the people imagining a future of superhuman intelligences have actually though through how it would actually work in the real world. We're going to spend trillions of dollars and vast amounts of resources chasing the goal of making ordinary humans obsolete. Now, that goal might be unobtainable, I hope. But I'm deeply alarmed at how much we're spending pursuing it.
Yeah, in my comment, I was assuming the publicly-stated goals of the labs actually came true. I assumed that they achieved true AGI (defined as "about as smart as Fable, except it can manage long-term tasks as well as a smart human, too"), and that this would cause widespread unemployment and fully-automated production. And furthermore, I assumed that once they could use AGI to automate more AI research, they could use that to make even smarter models. The labs call this "recursive self-improvement" (RSI) and like to insist it will happen Real Soon Now.
Given this chain of events come true, then I think there's an excellent chance that the resulting models will never be sold via an API. Given those assumptions, the labs would probably make more money by keeping all the compute to themselves and just ordering the AI to start and manage businesses. Imagine Anthropic having an in-house OpenClaw that could plan and run a successful startup with no human input besides an annual "board meeting" with the humans.
This isn't the only possible outcome, of course.
- Maybe the labs are wrong and progress stalls out well short of superintelligence. This would be nice!
- Maybe true AGI is only requires one or two clever algorithmic tricks beyond what we have now. In this case, training costs might go down, and superhuman models might become widespread. I suspect that possible future would be extra weird.
For example, I could imagine a future AI telling me that it has modeled my behavior and built a very large differential equation that seems to perfectly fit my ideal pattern to maximally achieve my life goals and it looks like something Ramanujan came up with, and I'd tell it "that looks great, let's optimize my life based on that" while having no ability to even approach understanding something like that.
Pretty sure it can dumb it down to cave man speak for any concept
Of course it would be great to make use of AI to solve cancer or fix other intractable problems, but we all know this isn’t the way things are going to go. The cancer is in our minds, our societies, and our norms that push us deeper and deeper into a grow-at-any-cost reality where the need for productivity is neither questioned nor considered in any real way. They say: we must grow! They say: we must be more productive! And we sit around thinking about who is going to control the productivity instead of acknowledging the real issues at hand.
I can only imagine a solution where we can all collectively agree that enough is enough. I’m not hopeful it’s possible and I think it’s probably the only way.
None of it is wrong exactly, but it feels like same enterprise-security machine finding the next anxiety surface than a "world is on fire right now" concern.
All of it always ends as a priesthood and a six-figure governance platform, rather than just taking practical steps to improve process.
Skynet has won.
There was a point in time when the majority of people were basically required to work on producing bare necessities like food. But we have already come far past that, where we could easily produce more than enough food for everyone on earth with a relatively small amount of workers.
There is some work that is still more or less essential to a healthy society that humans must perform, but many many people are doing something non-essential (and those doing essential work are often not rewarded proportionally to the "essentialness" of that work). We invent different kinds of "work" to fit into the capitalist machine that we have built, not because it is required for human sustainability or enriching our lives in any way. Some work might be actively harmful to society in every way other than keeping capital flowing around in some circle, and more and more of that capital is being captured/hoarded by some ultra wealthy individuals or corporations and not recirculating at all.
The problem is allocation. We allocate everything via capital, which I recognize has had some very positive outcomes at certain points in history, but may be reaching a point of making less and less sense for the modern world. The work I do honestly does not contribute much of anything to anyone. I do it because I am paid to, and I use that money to pay for my families needs. If my job was automated I would not be freed up for leisure, I would need to find some other work or service to be paid for to survive, regardless of how "important" that work is.
AI eliminating a swath of non-essential work does nothing to help with that allocation issue, in fact it probably ends in a worse overall allocation. AI may legitimately assist with some slice of "essential".work, but probably not that much of it.