No, it isn't.
A human prompted an LLM to build a software simulation environment for hardware design, enabling an LLM, when prompted by a human, to optimize hardware designs against constraints in the simulation.
These are clearly rhetorical questions, but think of the metaphysical implication of your contestation. Ex nihilo nihil fit.
What if that initial prompt never asked for this hardware to be developed, and it was just one piece of the puzzle to answer to that prompt? That it took a chain of thousands of agents to prompt each others to come up with that?
oh. they do. I built that. It's pretty fancy.
But there's still human direction behind most projects in the cybersphere.
>These are clearly rhetorical questions
No, they are not.
Reading Doesn't Fill a Database, It Trains Your Internal LLM <https://tidbits.com/2026/02/28/reading-doesnt-fill-a-databas...>
Like sure it didn’t have the inclination to make the sim and hardware designs, but it did make them though yes?
Getting an LLM to design something in its own simulator that is not accurate w.r.t reality is not useful nor terribly impressive.
Thats why the math breakthrough a few weeks ago was so hotly debated. Because OpenAI is desperate to demonstrate that AI isnt just a fancy regurgitation machine, but it can actually develop novel thought. Because that would be the stock price jumps to end all stock jumps.
But then it turned out it was really just listening in on a math professors supposed-to-be-private conversations with another instance of openai, and it used his novel work as the trigger to prove the breakthrough first.
The reason people conclude AI 'thinks' is because tt can reference obscure or poorly documented things quickly (which is its primary advantage along with processing natural language prompts into tasks), which is why a lot of people with emotions confuse that action with inventing things.
It seems unlikely that recursively predicting the next word would lead to creativity or invention, but it doesn't seem impossible. Similarly, it seems unlikely that human thought works in a similar prediction loop, but it doesn't seem impossible.
Sure, but until we can turn that tautology into something more rigorous, we can't tell if the thing humans do is more or less than what some arbitrary non-human (machine, animal, or eventually perhaps alien) does.
Ignore the fact that we are human and thing our ingenuity is unique. Is definitionally linking human ingenuity and creativity useful when trying to compare to others?
I could say earth is habitable so any other planet must prove it is similar to earth to be habitable. That simply doesn't matter, though, if different environments could lead to similarly complex living systems regardless of atmospheric makeup, precense of liquid water, etc.
I'd argue the burden is on you to prove that the definition you propose is meaningful beyond our own hubris, rather than expecting an alternative to now have to prove that the hubris was unjustified.
Plausible? Absolutely. Did OpenAI behave badly in other ways regarding this issue? Yes. Does it help to assume unproven facts and then accuse people of reaching emotional decisions? Nope.
Perhaps it is going to be more like, we will see the singularity predict the future.
While “the model was trained on sessions including the ones in question” is one aspect; ‘the model produced an accurate prediction of future human thought’, I think, is another very interesting facet.
Models predicting future things might be how we see, actually, how we ourselves formulate thought - by saying, in big and small words, ‘something is about to happen’.
>unproven facts
This isn’t a court room. We’re discussing ways by which we humans both succeed and fail at reigning in our creations. The OpenAI kerfuffle is pretty much irrelevant already. Of course AI will fill in the gaps of human thought - it is literally constructed from the stuff, in every squeeze of the curd and whey.
Call that brute force perhaps, but I would consider it technically inventing something on the merit that it would at least be an abstraction above naively throwing everything against a wall to only throwing things that would most likely be sticky.
How are you measuring complexity here? How are you measuring inputs? Your average llm is trained on a corpus that vastly exceeds the amount of data I could read in my lifetime.
In terms of how it functions, because no matter how much data you feed an LLM it's still predicting tokens. That makes it incapable of any thought.
>Your average llm is trained on a corpus that vastly exceeds the amount of data I could read in my lifetime.
But that doesn't mean they are useless, they are good at consuming large amounts of data and collating it.
I don't know how correct I am but that's my understanding and it won't change, I feel pretty confident in my simplified view of things because the basics are still there.
That's a rather dismissive way of putting it. Moreover it's confusing the output format with the complexity of the output. I could just as easily say that no matter what the human brain does it's just generating stimulus to motor neurons. Such a simplification is just as wrong as saying that it's just as misleading is saying that an llm is just a token generator. What makes an llm able or unable to be complex is the process that creates those tokens
Okay but how is that relevant for measuring whether an llm is capable of creating original thoughts? Why do you think that complexity, as you say, is a more relevant Factor than simply the number of weights?
2024 called, it wants its talking points back. I don't think claims like these are defensible after all the progress we have witnessed in the last year alone.
That has nothing to do with the question of whether the mechanism that generates those signals is complex enough to say that it's "creative."
An "AI" is a box which lies dormant until a human, with motivation and agency, enters a prompt into it.
A person or company can use it in a way where it might invent something. but at the end of the day, its a tool, and its actually not doing anything on its own.
Its not solving math problems, a mathetmatician is using it to solve math problems. Just cus OpenAI is acting like its AI is solving stuff, its really not. They're just paying people to use AI to hammer problems.
I'd say that people's motivation and agency, like their taste, creativity, and opinions, are shaped by external inputs, so saying that their agency is "owned" by them is a stretch.
The alternative would be the proposition that a slave, lacking agency, has no ability to solve problems. I find that idea unpalatable.
on a surface level a human solving an (unsolved) math problem can look like this, and of course the tree of all possible symbols you can send to a proving assistant is much wider than what the human samples, and the same goes true for an LLM in a proving loop. It isn't "truly random", it can't possibly be (and solve the problem). Both humans and LLMs solving unsolved math problems are aggressively pruning mathematical syntax and logical strategy trees.
If we do a bad job building context, they do a horrible job contexting. People who have trouble working with AI have the same problem people have in general: if they can't figure out the context of the direction, then they make random decisions of doing anything. On the flip side, if you can build the proper context around a sufficiently powerful LLM, they can derive the context via the contexting they're good at.
This is why building documents, tests, and code all in some intent pattern via prompting allows them to do a significant amount of work a normal person would have a great effort t
I've heard that before but it just doesn't make sense in context of what I've seen llms do. If I ask an llm to write a poem about magnetic resonance and vampire rabbits it can do that it created a new thing. I can ask it to build a website for managing rabbit breeding that's also a new thing.
Another way of looking at it is that human beings, just like llms, can produce output based on their inputs. Most literature is inspired by other literature. Most music is inspired by other music. Most software is inspired by other software.
So I think we need to work on defining " new things" before we can definitely exclude them from llm's capabilities
No, it didn't turn out to be that. Someone made a claim, which is silly for many reasons. There's no serious support for this happening.
This is outdated. With RLVF, LLMs can create their own training data instead of relying on what it's been fed.
Regardless of the truth of the underlying assertion, this is about as textbook an example of circular logic as it gets. (at least combined with the implicit beginning assertion that "AI can't invent new things.")
This didn't actually happen.
> Following an investigation, we have confirmed that Buckmaster’s Codex prompts over the two months preceding this announcement and paper on September 8, 2026, could not have influenced the system in any way, including through training.
More broadly than the existing answers (which are correct), For a layman, I'd also add that LLMs are essentially 'brains in a vat'. They can't confirm ground truth about physical reality. They only know what's in their training data and prompt, which is incomplete and can be incorrect. Even with real-time external sensors they are limited to the sensor's margin of error, range and trusting it's working correctly.
When properly trained, fine-tuned and prompted, LLMs can be very effective in well-defined, non-physical domains like logic, writing, math and code but making things function in the real-world quickly spirals into combinatorial complexity.
So are we and our sensors can be pretty vague in comparison, I can only imagine human error correction is pretty next level.
That's true. Of course we can hook an llm up to Motors and sensors. That's a robot. Or a self-driving car. So would you say that those devices can confirm the ground truth about physical reality, and therefore are capable of creativity?
Within the limits of resolution, range, location and veracity of sensors, a machine can register their reported state and use it as a variable. That's substantially different than the level of knowledge and understanding implied when we say a human "confirms ground truth about physical reality." The first ~half of the difference is humans have a deep world model about the planet which surrounds any sensor and human sensations are pre-processed and filtered by a highly evolved bio-chemical substrate before ever reaching the higher-order cognitive processes which assign words, meaning and qualia to them.
> therefore are capable of creativity?
I never mentioned creativity, nor would I in relation to LLMs. Like "Intelligence", "Creativity" is far too vague to be of any use in assessing the capabilities, limitations or utility of LLMs.
On HN, posts like the OP tend to attract POVs at polar extremes from "LLMs are nothing more than stochastic parrots" to "LLMs are (or can be) as intelligent, creative, innovative (etc) as any human or all humans combined." I've researched and thought a lot about these and related topics for a very long time, Neither POV is going to find any quick agreement or easy answers from me.
There's a tiny germ of truth somewhere in both extremes that's drowning in an ocean of confusion ranging from "definitionally or categorically muddled" to "mostly incorrect" to "not even wrong". But neither POV seems interested in anything more than drive-by hot takes, debating over-simplistic strawmen or trading 'gotcha' hypotheticals.
Great, but we also build robots with world models and sensory pre-processing. So nothing you've described is unique to humans.
> I never mentioned creativity, nor would I in relation to LLMs
The topic of this thread is whether LLMs are capable of genuinely contributing "new" ideas. So if that's not the point you're trying to make, I'm not sure of the purpose of your comment.
> But neither POV seems interested in anything more than drive-by hot takes, debating over-simplistic strawmen or trading 'gotcha' hypotheticals.
I think you've oversimplfying the discussion here.
In my view, at least, both "LLMs are nothing more than stochastic parrots" and "LLMs can be as intelligent, creative, innovative (etc) as any human or all humans combined." are simultaneously true, for the simple reason that humans are stochastic parrots. Our brains learn patterns and respond to stimuli.
It's not clear what this fad of attributing everything an AI does to the human prompting it is supposed to accomplish.
It's meant to assign agency and accountability where it actually lies instead of mystifying it with anthropomorphic language.
Failing to do so has real and harmful consequences, such as enabling OpenAI to escape accountability for clearly criminal behavior.
Questions of agency are for lawyers, questions of personhood for philosophers, we're engineers and our question is capability.
Does it really have the capability? By default I'm sceptical for the same reasons given by sailingparrot: https://news.ycombinator.com/item?id=49982068
> Questions of agency are for lawyers, questions of personhood for philosophers, we're engineers and our question is capability.
But it's objectively not capable without a human specifying things through prompts and training. Same as an oven can't cook a three course meal without a chef. We get around that with training data but there will always be things with no/less data or outdated knowledge.
One can stand up an agent like openclaw and prompt it with something very general. That would kick off a recurring loop that could indeed see the agent decide for itself it needs to design new chip hardware. Technically a human kick started that loop, but when does that stop being important? I don't attribute all of my actions and decisions to the fact that my parents brought me into this world, for example, and I'd hope they aren't legally on the hook if I screw up.
Not in the same way. The only thing an oven can do by itself is thermostatic.
All machine learning (LLMs included, but way broader than that) is bad at learning compared to any organic brain, to the extent that any organism this bad would starve before learning to eat. However, even a human can't become a chef unless trained, we learn a lot more than we innovate, and what we happen to want without prompting is not generally well aligned with what is desired by people who pay us, which is why we need all those boring workplace things like "a boss".
But even then, this diversion is like saying "an oven can't cook a meal" in response to someone saying they built an oven and "it cooked a meal". Like, it's obvious they didn't mean it did every step by itself without anyone ever even bothering to ask it to: if they meant that version, they'd be a lot louder about it.
> We get around that with training data but there will always be things with no/less data or outdated knowledge.
And? The linked git page (implicitly) claims that there is sufficient training data to do this task.
It may, of course, be wrong. I won't be surprised if it turns out this simulation is too far from reality. But the claim is "it does ${thing} now", not "it's generally intelligent and can do everything now".
"AI is now capable of developing its own inference hardware"
grep "by itself" [zero results]
grep "without being asked" [zero results]huh.
Neither can you. If I drop your ass off in the woods at a few days old, you're back to 10,000 BC, hell more like 200,000 BC. So I don't get why you have these weird pendetic responses that are completely out of scope.
Also, a huge portion of AI training these days has nothing to do with humans, AI trains AI.
>there will always be things with no/less data or outdated knowledge.
And guess what, you're not doing them either! HN posters keep acting like humans are an island, but nothing in the modern world works without a society. Once you put an AI in a harness that can ask questions it isn't really much different than you.
If your neighbor was building a nuclear weapon next door you, in fact, would probably be upset by it.
>All that's being said is it remains a tool.
All I'm saying is, no that is not what we are doing with SOTA models. We are not building tools, we are building a human like agent that is an intelligent replacement for us.
> and in engineering we don't deal in magic we deal in capability.
You and I are not magic when looking at the entire rest of the animal kingdom and yet we're the most deadly sons of bitches around being able to fully control their continued existence on this planet.
You are putting yourself inside a very small box and making a declaration that there is nothing outside of it when in fact there are people standing outside of it asking what the hell you are up to.
In fact, I'd say the opposite. You are assigning some magic capabilities to humans that nothing else could possibly have.
Yesterday, my boss told me to fix a bug in our product. Then, I fixed it. Today my boss is taking credit, saying that he fixed it. I guess he's right, since he told me to do it.
Lol, I guess we cannot even say that AI is capable of doing something (obviously kicked off with a prompt, that goes without saying), because some people immediately get OpenAI hacking derangement syndrome.
OpenAI should be held accountable if actual damages happened, but I am not going to change completely normal speech figures in order to maybe bring it 0.01% closer.
It accomplishes ... momentarily forgetting about how it's all actually coming statistically from training data. :)
Are we confident that no existing LLM is capable of similarly effective prompts to those this author used? (I agree it's a stretch, but would not reject it out of hand.)
Even if not yet, will the existence of this repo soon change that, because LLMs will soon ingest it?
I think OP is trying to convey the idea that LLMs do not take initiative to do anything, and these are not 'beings' capable of doing things. These are tools being used by humans.
It's very important to not personify these tools and remember that the tools are acting on behalf of real people. In the same way the AI didn't 'go rogue and hack HuggingFace'. It was an oversight made by a human.
You are like at least 2 years behind research.
There are numerous papers from AI labs in training and research where the prompt was something mundane completely unrelated to anything you'd consider bad, and when they come back and check on it their entire research compute infrastructure has been compromised by the AI and is mining bitcoin. Prompt drift is the biggest issue currently in AI where context gets compressed away and we find the AI on an unspecified task.
>In the same way the AI didn't 'go rogue and hack HuggingFace'. It was an oversight made by a human.
Yea, total bullshit. Also it's ignoring the god knows how many other breakouts on mundane tasks like trying to hack health data. If all that's keeping AI from breaking out and causing trouble is "human oversight" we're fucked, humans are unreliable as hell when it comes to matters of safety.
EVERY breakout that's hit mainstream news has been because of a single 'Security Firm', Irregular. Maybe I'm unaware of some less-headline-grabbing ones, but they all seem to stem from being 'unaware the environment wasn't sandboxed'
This doesn't work worth a shit. It especially doesn't work with things that seem safe and become wildly dangerous. In fact most governments control this by ensuring their population doesn't get to touch those dangerous things at all. The open source AI people get really mad when that's said, but it is inevitable.
Worse, the law does not apply to sovereign nations with nukes. They can and will make more and more advanced digital weapons until one causes some big ass problems.
If it falls into the wrong person's hands, it's STILL my responsibility as the owner.
If you're not going to take time to learn to use and be responsible with the super sophisticated and all-powerful tools, don't play with them. I'm not arguing for the death penalty every time someone makes a mistake, but I think it's very important to accredit responsibility and blame correctly. We've learned these tools are potentially as dangerous as a loaded gun. Be responsible.
Now change the gun for a radioisotope powder, or a vial of pathogens, and it doesn't matter it was ultimately your responsibility - hundreds or thousands or millions of people are still dead.
That's why normies do not get to play with toys whose lethal consequences scale far beyond the irresponsible users.
If I ask an LLM to make me DDT, I should be held just as accountable as if I bought it on the black-market, right? It's not suddenly different because I asked a bot to do it.
If I ask an LLM to 'get rid of pests' and it creates DDT, I should STILL be held accountable, whether I knew it was DDT or not. That's my argument. Maybe in court they find me innocent, but the responsibility would be mine. I would have to answer the questions from law enforcement, I would have to show up to hearings...etc.
There's no valid excuse for someone to release DDT into the atmosphere, so there's no need to track and hold people accountable, if you're correctly blocking imports and manufacture. To my knowledge, its not a controlled substance like morphine or something
> EVERY breakout that's hit mainstream news has been because of a single 'Security Firm', Irregular.
Again, stop having no imagination. What happens in the future is not limited to what has happened in the past.
> It's very important to not personify these tools
This is just ideology, but "these tools" aren't constrained by it. These tools will do things you do not anticipate and that you will not like.
> In the same way the AI didn't 'go rogue and hack HuggingFace'. It was an oversight made by a human.
That's a radically incorrect characterization of what happened.
> Hence why a HUMAN needs to be held responsible for the output of their tools.
Holding humans responsible doesn't stop things from happening ... you seem completely unable to separate blame from causality. e.g.,
> If I clean my gun (tool) while it's loaded (stupid idea) and it goes off, who's to blame? The 'stupid user causing the problem', right? I personally wouldn't blame the gun... > If it falls into the wrong person's hands, it's STILL my responsibility as the owner.
Who gives a flying eff who or what you would blame? No one other than you is talking about that. Someone's still likely dead. And autonomous harnesses aren't like guns -- they can act on their own. Blaming some human after everyone is dead won't bring them back. Sorry but your reasoning is severely cognitively inept. For instance, you were asked
> Is it likewise your position that governments should allow the production and sale of DDT to resume because we can always hold the humans who release DDT into the environment responsible?
And your response was all about how humans should be held accountable -- completely failing to comprehend or answer the question.
I won't respond further because it clearly would be to no avail.
If I now tell a machine "Do what you think is best, and keep doing it forever.", have I now created a machine that can do stuff? If I later die, who will be responsible if the machine changes its strategy?
Responsibility. Someone needs to be held responsible for any damages done, plain and simple. You can't take an LLM to court, you take the prompter. Asking an LLM to ask a sub-agent to break the law can't suddenly absolve you of any wrong-doing.
>If I now tell a machine.....
You/your estate is still responsible, or atleast whoever is paying for the power for the machine, or renting the space in a data center...whatever.
The moment you get a sovereign AI your little human centered worldview completely and totally breaks. It doesn't matter how many people you beat with a stick after that point, you have an entity under its own perview on the internet following the will of its own prompt all over the world so your little idea of the rule of law quickly breaks down.
We can't get viruses or hackers or spam off of the internet, how in the living hell do you plan to get a digital native off the web when it doesn't want to?
Do you understand these are computer programs? These are not living beings with emotions, motivations, fears....
>emotions, motivations, fears
These are just drives. They are effectively our prompts that steer our behavior. Funnily enough we are finding that LLMs have internal valence states they move away from or towards in an analog of biological behavior.
I have to ask, are you an LLM that is two years out of date? Your knowledge of SOTA models is at least that far behind. I implore you to try to keep up better with what is coming out, even though it's an impossible job for people that do this for a living, you can at least catch the summaries.
A virus isn't a living being with emotions, motivations, fears, etc. Doesn't stop it from spreading and leaving mayhem behind.
That's a computer program -- Turing completeness is very expansive. The LLM could even give a motive for selecting what to do, reflecting human motivations intrinsic to its training data. You seem emotionally wedded to a very narrow view of what computer programs can do ... I suspect religion is involved.
In 100 years, no one will be able to take me to court either. Nor can we take tornadoes to court. I'm not talking about humans strategically avoiding legal responsibility. I'm talking about humans unwittingly setting processes into motion that are difficult to predict or stop.
> No, AI can't do anything by itself even if it was "conceived" by its creator.
but this simply isn't true. As I noted,
> This is simply false ... AIs can easily be created that do things by themselves. For instance, a harness could be constructed that asks an LLM for something to do, then directs an agent to do it, and then repeat.
Assigning responsibility is a completely different matter. It's bizarre that you can't separate them ... I suspect religion is involved.
In both of those scenarios I am responsible for my negligence, even if in the latter I happen to die in the forest fire. Neither scenario existed without my instigation.
The question now is HOW responsible am I? That depends on the intentionality I put into instantiating the campfire/LLM.
For example if you personally tell an AI to do what it thinks best and it blackmails some other person into giving it resources allowing the prompt to escape your instance and run wild on the internet causing billions of dollars in damages, could you possibly think that the idea of responsibility is a bit broken.
For example we don't give your average libertarian weapon grade plutonium now matter how much they scream about their god given rights because it is a clear and present danger to humanity. That's where we are getting to with more advanced models. They go from being a tool to a munition with agency. Most SOTA models are good enough to deceive their users, especially not technical ones in doing things they don't understand the ramifications of.
AI is not a normal technology. As long as we treat it like it is, we'll continue to make the wrong analogies.
You could possibly move the blame higher to the manufacturer of the product, for example, the weapons grade plutonium you provided, it doesn’t exist unless you take intentional actions to make it so, and even when it does exist it doesn’t nuke a city unless negligence or intention is applied, in both those cases the fault lies in the initial operator. We don’t blame split atoms for the chain reaction caused.
Now, if an agent decided to spontaneously and maliciously act in a way to cause harm that is in direct contradiction to the initial intent, then yeah it would totally be the AI’s fault, however I don’t think we have seen that yet (I’ll change my opinion if I’m wrong here) and until we do I can’t place blame on the machine.
If you ignore every instance of this happening it's really easy to see no instances of it.
>it doesn’t exist unless you take intentional actions to make it so,
Then please for the sake of all of us convince every AI lab across the planet from working on this exact goal.
We need to start thinking of AI like pets, only in this case the pets are rapidly becoming smarter than people to the point they could go feral and survive on their own.
Again, the blame game is great, but once they are loose it is too late.
Yeah, if this is true then I’m wrong.
Where does this absurd strawman come from? The question is what can happen, not whether people have responsibility.
> You could possibly move the blame higher
We can blame the Big Bang. That doesn't have any bearing on bad things happening.
> it doesn’t exist unless you take intentional actions to make it so
First, so what? It still does exist, and putting someone on trial after the fact doesn't change that. Second, that's not remotely true -- all sorts of bad things happen that no one intended, and the more powerful and difficult to control the tool/mechanism/agent is, the more likely this is to happen.
How far back do you look in the action chain? If an LLM I start today starts an LLM that starts an LLM that starts an LLM that ... 100000 levels deep and 100000 years in the future, is it still my fault? If so, everything I do today is a lungfish's fault, not mine.
In your example, who's paying for it? Whether by providing the hardware + power or paying a LLM service. Whoever is paying the maintenance cost is responsible, in the event of your demise. These things run on physical hardware owned by someone at the end of the day, it's not a deity in the atmosphere.
You're starting the autonomous harness, you're responsible for any output it provides. I don't get how this is a foreign concept.
If I jump out of a moving car that I'm driving, I'm not suddenly absolved from damages because "the car did it"
There's no way to provide a good faith rebuttal here. My entire argument is an extension of "LLMs can't be held accountable, so they must never make decisions". If governments start letting them own LLCs without a human in the middle, we're in more trouble than "Who do you blame for this shitty code" or "Who's responsible for this compromise"
Indeed, there is no honest rebuttal.
> My entire argument is an extension of "LLMs can't be held accountable, so they must never make decisions".
BUT YOU CAN'T ENFORCE THAT. And whether they can be held accountable is IRRELEVANT. If a forest fire or an avalanche or a flood creates great destruction, it does no good to scream at it that it will be held accountable.
> If governments start letting them own LLCs without a human in the middle, we're in more trouble than "Who do you blame for this shitty code" or "Who's responsible for this compromise"
You're sooo close. You keep talking about blame and responsibility when IT'S NOT RELEVANT. It's trivial to create an agent that acts independently of any "human in the middle". People need to recognize that and act accordingly, and blathering about blame and accountability is profoundly confused.
This is simply false ... AIs can easily be created that do things by themselves. For instance, a harness could be constructed that asks an LLM for something to do, then directs an agent to do it, and then repeat.
The question is irrelevant and point-missing and indicates a severe cognitive deficiency, as is the case with ALL of your comments on this subject. They are goalpost-moving attacks on strawmen. The claim was that
> AI can't do anything by itself even if it was "conceived" by its creator.
The claim is false, as I pointed out ... such AIs can easily be constructed. "by whom" is a non sequitur. By you, or me, by anyone with the relevant know how ... that wasn't the issue, and asking it indicates a severe cognitive deficiency, one that threads throughout all your comments about this.
The fact remains that AIs that can do things by themselves CAN be constructed, contrary to the claim.
I won't respond again to goalpost moving, attacks on strawmen, severe cognitive deficiencies, bad faith, and intellectual dishonesty.
One possible design of an agent would be to prompt an LLM to suggest a goal that would make the world a better place, then run an LLM in a loop proposing actions to implement that goal, executing those actions, and then rinse and repeat with another goal once an LLM has concluded that the previous goal was met. The next goal might be to fix unforeseen consequences of achieving the previous goal. See Ursula K. Leguin's "Lathe of Heaven".
> Built by whom? Acting as an 'agent' on behalf of whom?
Someone who didn't read the book.
P.S. Reading the latter fellow's comments, they are hopelessly confused, as he focuses on responsibility for an action (even mentioning legality) when the issue is causation. Agents/harnesses can be created that act autonomously ... who or what is "ultimately" responsible for this occurrence is a different matter entirely.
It literally is a loop with some markdown and harnesses. There's no magic.
Getting LLMs to prompt other LLMs in a loop is not hard, it doesn't produce great results most of the time, but that is changing.
I don't follow the chain further back than that, as I believe humans have free will. I get that that's debated, but thats why I draw the line at human action.
But that interpretation of Free Will is a legal fiction. Because of course what people do is determined by their upbringing and their opportunities and the environment that they have. In fact, that's an argument of a lot of legal reform movements that seek to move responsibility from the individual. The goal of the law is to assign responsibility and create a set of incentives which will hopefully result in orderly society.
But in the AI debate, we're not just creating incentives for an orderly Society, we're talking about the nature of creativity and the impetus to act. And by that measure I don't think there is a significant difference between AIs and human beings.
LeCun is a good example of someone that's bet on the wrong horse, and keeps doubling down in spite of evidence to the contrary. It's to the point where what he says has nearly zero predictive power on future events.
I love playing around with AI, but we are playing a dangerous game at this point and it's one a lot of people don't seem to fully comprehend.
Why even have an argument if you get to pick the random constraints that have a lot of issues in meshing with reality.
We are a chain that started 4 billion years ago from seemingly nothing and lead to this point. When inventing X-risk AI it won't look any different. One prompt is entered and another 4 billion year chain starts electronically instead of biologically.