This guy is vibing some react app, doesnt even know what “npm run dev” does, so he let the LLM just run commands.
So basically a consumer with no idea of anything. This stuff is gonna happen more and more in the future.
We need local sandboxing for FS and network access (e.g. via `cgroups` or similar for non-linux OSes) to run these kinds of tools more safely.
In practice I just use a docker container when I want to run Claude with —-dangerously-skip-permissions.
sure, it would be amazing if everyone had to do a 100 hour course on how LLMs work before interacting with one
- sell a knife that can lead to digit loss, or
- sell software that interacts with your computer and can lead to data loss, you can
- give people software for free that can lead to data loss.
...
the Antigravity installer comes with a ToS that has this
The Service includes goal-oriented AI systems or workflows that perform
actions or tasks on your behalf in a supervised or autonomous manner that you
may create, orchestrate, or initiate within the Service (“AI Agents”). You
are solely responsible for: (a) the actions and tasks performed by an AI
Agent; (b) determining whether the use an AI Agent is fit for its use case;
(c) authorizing an AI Agent’s access and connection to data, applications,
and systems; and (d) exercising judgment and supervision when and if an AI
Agent is used in production environments to avoid any potential harm the AI
Agent may cause.It's perfectly within the capabilities of the car to do so.
The burden of proof is much lower though since the worst that can happen is you lose some money or in this case hard drive content.
For the car the seller would be investigated because there was a possible threat to life, for an AI buyer beware.
These are being sold as a way for non-developers to create software, I don't think it's reasonable to expect that kind of user to have the same understanding as an actual developer.
I think a lot of these products avoid making that clear because the products suddenly become a lot less attractive if there are warnings like "we might accidentally delete your whole hard drive or destroy a production database."
Google (and others) are (in my opinion) flirting with false advertising with how they advertise the capabilities of these "AI"s to mainstream audiences.
At the same time, the user is responsible for their device and what code and programs they choose to run on it, and any outcomes as a result of their actions are their responsibility.
Hopefully they've learned that you can't trust everything a big corporation tells you about their products.
LLM makers that make this kind of thing possible share the blame. It wouldn't take a lot of manual functional testing to find this bug. And it is a bug. It's unsafe for users. But it's unsafe in a way that doesn't call for a law. Just like rm -rf * did not need a law.
The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
> The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
That absolutely happens, and it still amazes me that anyone today would take at face value anything stated by a company about its own products. I can give young people a pass, and then something like this will happen to them and hopefully they'll learn their lesson about trusting what companies say and being skeptical.
Or just anyone non-technical. They barely understand these things, if someone makes a claim, they kinda have to take it at face value.
What FAANG all are doing is massively irresponsible...
... Except perhaps with phrases like "major company" and "for profit", and "not legally actionable".
Right here. And I think you're not quite getting it if you have to refer to "go on the internet and tell lies"...
Sure plenty of people might be on "social media" and have some idea that people fib, but they aren't necessarily generally "surfing the internet".
To them, saying "the internet tells lies" is comparable to saying "well sometimes, at the grocery store, you buy poison instead of food", and yes, it can happen, but they aren't expecting to need a mass spectrometer and a full lab team to test for food safety... to you know, separate the snake oil grocers from the "good" food vendors.
Maybe AI providers should give more warnings and don’t falsely advertise capabilities and safety of their model, but it should be pretty common knowledge at this point that despite marketing claims the models are far from being able to be autonomous and need heavy guidance and review in their usage.
Note how OP is very nonchalant at all the responses, mostly just agreeing or mirroring the comments.
I often see it used for astroturfing.
Not knowing is sort of the purpose of AI. It's doing the 'intelligent' part for you. If we need to know it's because the AI is currently NOT good enough.
Tech companies seem to be selling the following caveat: if it's not good enough today don't worry it will be in XYZ time.
> It still needs guardrails, and some domain knowledge, at least to prevent it from using any destructive commands
That just means the AI isn't adequate. Which is the point I am trying to make. It should 'understand' not to issue destructive commands.
By way of crude analogy, when you're talking to a doctor you're necessarily assuming he has domain knowledge, guardrails etc otherwise he wouldn't be a doctor. With AI that isn't the case as it doesn't understand. It's fed training data and provided prompts so as to steer in a particular direction.
Frankly, having a space in a file path that’s not quoted is going to be an incredibly easy thing to overlook, even if you’re reviewing every command.
While using the vibe coding tools it became clear to me that this is not something to be used by folks who are not technically inclined. Because at some point they might need to learn about context, tokens etc.
I mean this guy had a single window, 10k lines of code and just kept burning tokens for simplest, vague prompts. This whole issue might be made possible due to Antigravity free tokens. On Cursor the model might have just stopped and asked to fed with more money to start working again -- and then deleting all the files.