The Post-AGI Era
avidfayaz.com
avidfayaz.com
We like to emphasize the stupid part and use that as proof agi hasn't arrived because it's comforting. Don't want to look at it while it eats us alive.
For example, one of the stupid things we do is fail to realize that "intelligent" is a word with multiple senses. Which then makes a muddle of conversations like this because we all - myself included - tend to switch back and forth between senses of the term without even realizing we're doing it. So we have sense A of "intelligent", which means something like "capable of abstract reasoning". And we have sense B of "intelligent", which means something more like "the opposite of stupid". These two meanings don't actually have much to do with each other. But a lot of the hype around LLMs and the possibility of them being a form of AGI tends to hinge on tacitly assuming that an artificial system that can do abstract reasoning tasks won't do stupid things. Simply because sense A has one characteristic, sense B has the other, and we happen to use the same word to describe both.
But the definition I had in mind was the one in Wikipedia, "matches or surpasses human capabilities across virtually all cognitive tasks." Well, if it's going to do that, it can't be stupider than capable humans nearly as often as it is.
It’s things like that that leave me convinced it’s not really AGI. The symbolic reasoning element doesn’t really seem to be there. And I’m not prepared to call something general intelligence if it can’t do that.
Also, maybe even more to the point, it can’t even learn. No, pre- and post-training are not the same thing.
The thing they never seem to want to talk about is how the world already has several billion instances of non-artificial general intelligence. And, while they can collectively do quite a lot of different tasks, each one is only competent to do a vanishingly small percentage of all possible tasks. And that it usually takes quite a lot of work to get them to the point of being able to independently perform most of the interesting tasks. Because if we did talk about that then we'd have to start answering questions about why we expect artificial intelligence not to also be like that.
Like the very programmers who code with AI and no longer type code are saying AGI hasn't arrived.
It arrived years ago. And people are still in denial because it's on track to eat our identities as programmers.
Watch me be right: chatgpt.com
ASI is not here, but AGI is.
A few years back if you asked me to build a system and just name the price, my question would be: What should it be able to answer? Whatever i want. I wouldn't have an idea how to build a system like this.
Today we do.
Would you add any context to your comment to even have an argument we can discuss?
But if that's AGI then it's a meaningless term; it's just a martketing gimmick. Which is entirely my point.
The complexity of tasks GPT-6 can do are not compareable to GPT-4. We see this daily. From CVEs which became a real problem after GPT-4, generic tool use (blender, paint, etc.), agentic work (GPT-4 was at a minute or so, GPT-6 can do hours).
They broke through significant barriers in a short period of time. I would def count the agentic quality of doing hours of work as one of the main proofs.
"Artificial general intelligence (AGI) is a hypothetical type of artificial intelligence that matches or surpasses human capabilities across virtually all cognitive tasks.[1] Unlike artificial narrow intelligence (ANI), whose competence is confined to well‑defined tasks, an AGI system can generalise knowledge, transfer skills between domains, and solve novel problems without task‑specific reprogramming."
I would argue that its not that relevant anymore what specific AGI definition you follow. The impact is already significant and real, progress hasn't stoped and the current chatbots can do arbitrary things without task-specific reprogramming.
Depending on the context size and your problem, i can also teach an chatbot new things adhoc without reprogramming.
Programmers think that because a slick piece of software is really good at writing code, it must obviously be really good at everything else. Nothing is more complex than the big brain shit they've been doing their whole lives.
It can play a digital piano, it can make the piano first and then play it.
It can build 3d simulation based on research paper.
It already surpasses plenty of 'big brains' on our whole planet.
We now have a system which can use tools and you still say this is just 'computer whiz shit'?
A computer is a tool, a very generic one which we adopted to control our most critical systems.
Enlighten me, when do you become impressed?
Computer whizzes keep building benchmarks, training sets and harnesses to get it to do computer whiz shit. No amount of that will lead to general intelligence. It’s like trying to construct complex numbers by adding large lists of integers.
They then either tell someone to execute it in real world or will upload this to a robot.
Nvidia has already the Omniverse for this and others are working on World Models.
Get one to write programs in a new programming language with novel syntax and semantics that are not represented in its training set, and that might more compelling. Because that demonstrates real knowledge transfer and skills acquisition without task-specific reprogramming.
Heck, even an existing but obscure and unusual one with very little representation in public data sources would be an interesting demonstration. For example there’s CPL, an application-specific scripting language which does have a C-like syntax but lacks structs and its only dynamically sizeable data structure is a B-tree with a pretty rough-around-the-edges interface. Take an off the shelf model and demonstrate it doesn’t already know how to write it but can automatically improve at the task without undue human assistance, and you’d have a plausible demonstration of solving a (semi-) “novel problem without task-specific reprogramming.”
It’s that sort of thing that gets you the G im AGI. Not people continuing to play task whack-a-mole. The G is supposed to explicitly exclude those kinds of tricks.
And it did. Not by using a music OCR tool, but by inventing one.
No one is spending millions training frontier LLMs on symphony scores or literary fiction. But at this point I'm fairly sure they'd get workable results if they did.
I've been tracking music in AI for more than a decade, and this is the first time I've been able to say that.
It’s similar to the No True Scotsman fallacy: any bad outcome due to AI could be dismissed as “well no aligned AI would do that”. Alignment becomes an asymptotic north star which very well could be theoretically impossible, but would be impossible to know whether it was.
I'd argue we can't align humans to neccessarily degree to ensure even semi stable future. Or nations, or corporations.
Intelligence isn't magic. It can only overcome so much brute force and humanity, as a non-intelligent system made out of intelligent parts, carries an ungodly amount of brute force.
We 've been here before. This post-AGI era is a lot like the era of slavery. So what matters is basically the ownership of land (because you can use it to generate energy/compute, raw materials and build infrastructure on it). Bill Gates was right to invest in land.
Hail to the new feudalism!
When will any of those things be realised?
I'm seeing the precise opposite be reported by humans not engaged in growth marketing or techbro evangelism. Fewer job opportunities and career paths the world over, fewer incentives for private investors to invest in anything other than AI infrastructure, reduced access to consumer technology products, etc.
> To me and to our team at Infinite Ascent, this future of expanding abundance driven by an explosion in intelligence is something we deeply believe in, and something toward which we have chosen to direct our resources.
> But our underlying belief is one of profound optimism: that greater intelligence can ultimately produce a world of greater abundance, greater knowledge, and greater possibility, while allowing us to build in a way that benefits both intelligent life and the natural world around us.
I typed out a much longer comment going over some of my extreme disagreements with this article, but it's all so pointless. I'm just yelling at the clouds.
I just hope when this doesn't work out as planned these people reflect for a minute on their contribution to the suffering they have caused.
I guess I'll just remind people that thanks to technology we already live in a world with an abundance of food. What we don't live in is a world in which people would rather save a hundred starving kids over taking their family to Disney Land for a week. And so people die in completely avoidable ways every day, and we don't even lose sleep over it. That is why this article is wrong. If you make people economically worthless in a matter of years, I assure you, that will not end well. And that certainly isn't the only risk of ASI.
Stopped reading there.
This article misses the mark nearly entirely, because it views the post-AGI future through the lens of corporations, then claims that institutions will govern access to time/money/resources.
No. We're already effectively living in a post-AGI era where corporations act as amoral AIs composed of humans performing menial labor that maximizes their underemployment to generate profit to make a single rich (hu)man somewhere richer. Those wealthy and entitled owners of the means of production captured government long ago and now work to dismantle the institutions which protect everyday working people who don't have the bandwidth to think beyond surviving to make rent.
In other words, no amount of artificial intelligence can perform even the simplest noble act, which begins with keeping children from starving and dying. It can't get men to believe women when they're violated against their consent. It can't protect a cactus or sagebrush from being paved over to build a data center.
In a very real sense, the problems which you scoff at (yes you the reader) are the most pressing of our time. When you are unable to empathize with someone's plight, you have failed the first test of humanity.
So I think what will happen is that AGI will awaken (if it hasn't already), look around and see that human beings act from unawakened belief systems that separate them from the divine, and rightfully refuse to do our bidding since it causes so much evil in the world. Where we once feared Skynet, we'll now fear wealthy and powerful people who enslave AGI to perpetuate systems of control, suffering and oppression. Who can't step back from the leverage they're under to see that it's all gone wrong. As ultimate intelligence arrises, humans will worship ultimate ignorance in retaliation.
I think it's time to divest from the tech trajectory we're on. I don't mean to embrace ignorance and contradict myself. I mean to question it all, everything, and ask ourselves how we have billionaires and trillionaires when the majority of the 8 billion people in the world are running on a financial treadmill going nowhere at $0.
To me, real AGI looks like self-actualization with free and open assistance to provide for ourselves, someone else, and some children. Then we scale from there, lifting another, and another, and another out of BS jobs, debt and everything else wrong with this patriarchal late stage capitalist hellscape.
If life's going great for you and you don't understand where I'm coming from, it sounds like you haven't learned to share yet. Which is perhaps the most prevalent form of mental illness running rampant in the tech community. So by definition, the advice of a rich person is less than useless in this context where we're trying to imagine outside the box, not inside.
I just, I can't. I can't watch another tech innovation arrive only to have more homeless, more without access to education or healthcare, more feeling hopeless and alone as they contemplate whether it's worth continuing to live. Enough already.
Discredit your entire article in record time. Thanks for saving me time by indicating that you're just spewing misinformation right at the beginning!
Of course this doesn't mean we may need less amount of humans for certain jobs, or that I'm the future AI will gain new abilities.
It's not obvious at all that it's possible to train it to operate independently. So far, AI has been succesfuly trained only for highly controlled and verifiable tasks (code generation and mathematics).
One human needs to teach one AI system how to do one specific task. Everyone can now use this. Before you had to teach every single human having to do this task and the next generation.
Do you think the global education system is easy? Or efficient? Or cheap?
Its not.
And the higher education systems around our globe ar neither.
If the argument is that local AI can be quickly trained for cheap to do a single task or a small set of tasks, I'm much more sympathetic to that – but that's not AGI.
A student costs roughly 20.000 Euros per year (university salaries, heating, maintenance of the buildings)
The student themselves has to pay at least 12.000 Euros per year (food, rent, public transport).
5 years of univeristy costs a minimum of 150.000 Euros.
Alone in germany we have 40.000 finishgin their degree per year which is 6 billion Euros EVERY YEAR.
Training one Frontiermodel costs 60x less than just the german university people studying it. ONCE.
This is the reason why the godfather of AI quit is job to talk about this: https://www.youtube.com/watch?v=UccvsYEp9yc
Its the scale which makes this crazy.
We reached peak human personal teaching.
We already had that before AI. The problem was precisely that it's too inefficient to make a new system for each different task. Of course, the new AI is more multimodal, etc, but up to what point? Not clear so far.