Go ahead and try it with your favorite LLMs. They're too deferential to push back consistently or set up a dialectic and they struggle to hold onto lists of requirements reliably.
[edit] In other words, llms that lie less will be more valuable for certain people, therefore llms that tell you when you are dumb will eventually win in those circles, regardless of how bruising it is to the user's ego.
Even experienced engineers can be surprisingly bad at this. Not everyone can tell their boss “That’s a stupid requirement and here’s why. Did you actually mean …” when their paycheck feels on the line.
The higher you get in your career, the more that conversation is the job.
They just want a yes-bot.
I think that the more you find ways to (productively) make that conversation your job, the higher you get in your career.
There's also no "ruling out" the Earth will get zapped by a gamma-ray burst tomorrow, either. You seem to be talking about something that, if done properly, would require AGI.
You can do anything with AI. Anything at all. The only limit is yourself.
Nobody can predict how soon a technology will plateau. People make predictions based on insane hot takes like "I pray to our new overloards who will make me immortal" and "there's nothing new under the sun, everything is just marketing"...
There might be the next AI winter starting starting next year, AI might wipe out humanity in our lifetime, even both of those might happen. Both very unlikely (qualitatively) ends of a huge spectrum.
That's not to say that we "don't know anything and thus give up talking about it". Otherwise I wouldn't participate in this discussion and I hope you wouldn't either if you didn't expect to sometimes learn something or be confronted with a new idea.
I just find both attitudes "it's making us immortal" as well as "I'm so experienced, I know that no new technology ever lives up to expectations and can mock people who admit they don't know that" unproductive. You don't know. Most technologies don't live up to expectations, a few work out even much better than expected. I'm sure before the Lee Sedol match, you thought to yourself "right, so it's going to win this and then plateau exactly a decade later"?
The thing is, with enough magical thinking, of course they could do anything. So that let's unscrupulous salesmen sell you something that is not actually possible. They let you do the extrapolation, or they do it for you, promising something that doesn't exist, and may never exist.
How many years has Musk been promising "full self driving", and how many times recently have we seen his cars driving off the road and crashing into a tree because it saw a shadow, or driving into a Wile E Coyote style fake painted tunnel?
While there is some value in evaluating what might come in the future when evaluating, for example, whether to invest in an AI company, you need to temper a lot of the hype around AI by doing most of your evaluation based on what the tools are currently capable of, not some hypothetical future that is quite far from where they are.
One of the things that's tricky is that we have had a significant increase in the capability of these tools in the past few years; modern LLMs are capable of something far better than two or three years ago. It's easy to think "well, what if that exponential curve continues? Anything could be possible."
But in most real life systems, you don't have an unlimited exponential growth, you have something closer to a logistic curve. Exponential at first, but it eventually slows down and approaches a maximum asymptotically.
Exactly where we are on that logistic curve is hard to say. If we still have several more years of exponential growth in capability, then sure, maybe anything is possible. But more likely, we've already hit that inflection point, and continued growth will go slower and slower as we approach the limits of this LLM based approach to AI.
And what is the endgame for AI?
Also technologies don't develop like evolution really so not sure why you drew that comparison.
Technologies plateau for a combination of reasons - too expensive to make it better, no interest in making it better, can't figure out any more science (key people involved leave / die / lose interest, or it's just too difficult with our current knowledge), theoretical limits (like we are reaching in silicon chips). I don't see a lot of similarity with evolution there.
This is what people never understand about no coding solutions. There is still a process that takes time to develop things, and you will inevitably have people become experts at that process who can be paid to do it much better and quicker than the average person.
It applies outside of tech too. Even if you can make potato pave at home, having it at a restaurant by someone who has made it thousands of times everyday, is preferred. Especially when you want a specific alteration
The out put was verbose, but it tried and then corrected me
> Actually, let me clarify something important: what you've described - "every point equidistant from the center" - is actually the definition of a circle, not a square!
here's the prompt
> use ascii art, can you make me an image of a square where every point is equidistant from the center?
If we're just looking for clever solutions, the set of equidistant points in the manhattan metric is a square. No clarifications needed until the client inevitably rejects the smart-ass approach.