roywiggins

This article is certainly substantially AI written.

And you don't need to trust Pangram (I think Pangram is pretty good, but). Imho the headings alone are sufficient to give it away, Claude et al love "The XYZ Dilemma" etc and that's every single one of the headings here.

"With your permission, Copilot may execute code or bash commands in this folder." could be interpreted either way I suppose, but the actual question is "do you trust the files in this folder" and not "do you trust Copilot to execute any bash commands it wants without further permissions prompts".

The risk isn't solely that there might be a prompt injection, Copilot could just discover `env sh` doesn't need a user prompt and just start using that spontaneously and bypassing user confirmation. If you haven't started Copilot in yolo mode that would be very surprising and risky.

If it usually asks for user confirmation before running bash commands then there should, ideally, not be a secret yolo mode that the agent can just start using without asking. That's obviously a bad idea!

"Actually copilot is always secretly in yolo mode, that's working as designed" seems like a pretty serious violation of expectations. Why even have any user confirmations at all?

It's because in this case "curl" is just a parameter to env. Env just happens to execute curl (or indeed sh, which seems, uh, worse).

Seems nuts to have env or find on the default allowlist to me! Really these agents shouldn't be able to execute anything at all without approval by default, if you want to give it something like "find" or "env" to do safe things without approval, reimplement the functionality you want as a tool that can't do arbitrary code execution.

> general-purpose computing device

This term doesn't seem defined in the law at all. How general is general?

Graphing calculators that support apps and Python? Of course, they don't usually have "accounts" either. But to a technologist it's a "general purpose computer" insofar as it can run new code that the user loads into it, it can definitely run games that it didn't come from the factory with, etc. It's a tiny multipurpose computing device.

The user didn't need to give it curl permission, that's the whole issue:

> Copilot also has an external URL access check that requires user approval when commands like curl, wget, or Copilot’s built-in web-fetch tool request access to external domains [1].

> This article demonstrates how attackers can craft malicious commands that go entirely undetected by the validator - executing immediately on the victim’s computer with no human-in-the-loop approval whatsoever.

It's probably bad that the system 1) usually prompts you to take shell actions like `curl`, but 2) by default whitelists `env` and `find` that can invoke whatever it wants without approval.

If 2) is fine then why bother with 1)? In yolo mode such an injection would be "working as designed", but it's not in yolo mode. It shouldn't be able to just do `env sh` and run whatever it wants without approval.

Even the title follows a common LLM pattern (The X Problem/Issue/etc).

I don't know how people read this sort of LLM output without their eyes glazing over and tuning it out. Every blog post authored or substantially edited by Claude sounds the same sort of vaguely pompous and stilted, surely people are bored of it by now? But apparently not.

One example of being "too structured" is that LLMs love an explicit introduction and conclusion even when one that isn't really warranted. It's always telling you what it's going to say, and what it just said.

I've reached the point where if any blog post has a subheading with some variant of "The Problem", I assume it's been edited with an LLM, because it co-locates with other indicators so strongly.

Firefox translation is entirely offline, so is the LLM powered tab groups. Nobody is making money off Firefox Translations.

Or just verify and write up its findings yourself, this is like pasting notes from a research assistant in verbatim. It comes across as pretty lazy!

Pangram[0] thinks the closing part is AI generated but the opening paragraphs are human. Certainly the closing paragraphs have a bit of an LLM flavor (a header titled "The Pattern", eg)

[0] https://www.pangram.com

Airplane "autoland" goes back a ways:

https://en.wikipedia.org/wiki/Autoland

That's because it's llama3 8b.

From my point of view, if I wanted an AI summary of a project I could generate one myself. An unlabeled AI readme is almost worse than nothing! I've generated AI readmes myself- they can be useful- but they aren't something to show off.

I'll read a badly-formatted readme written by a human with far more interest than a formulaic LLM summary of a project. But it seems like nobody even notices a readme is slop because it has nice Markdown, and my best guess as to why is that people have become habituated to this stuff.

Darren Aronofsky

https://www.decodingeverything.com/darren-aronofsky-ai-slop-...

(Also, this website when Show HNs with slop READMEs get to the front page and nobody seems to notice that it's written in grating Claudese.)

Here's another thing: I think spending too much time with generative AI makes your taste worse, by habituating you to stuff that's pretty bad.

I think it's a sort of slot machine effect, you get used to losing and when something goes slightly well you wildly overestimate how good it is. You see this with visual artists who got way too into image generation. Because they have to spin the wheel a thousand times to get one good output, they have totally habituated themselves to a lower standard by the time they emerge from the AI mines clutching their one good output, because that output is not all that great.

It looks good compared to all the failed generations though!

Also, spending all your time cranking the slot machine handle and occasionally winning convinces your brain that you have a magic ability at cranking the slot machine handle, when actually you were at best slightly lucky. So you get people who convince themselves they are geniuses at using AI when they are actually average or slightly above average.

The Nine Quintillion UUIDs of God

gotta be careful:

https://hex.ooo/library/nine_billion_names_of_god.html

Equally, the idea that there is a tradeoff in some particular situation is frequently asserted without evidence. The quote from the article is "It is always the case that there are benefits available from relinquishing core civil liberties."

That is 1) presented without evidence and 2) almost certainly false. It is not always the case.

> There's no government as effective at instituting necessary changes as a benevolent autocracy

Autocracies have lots of issues around eg building a sufficiently capable bureaucracy that isn't too corrupt to do things. It can make it harder, not easier. Democracy can lean on democratic legitimacy, constitutional traditions, and a history of allowing power transitions without anyone losing their heads or launching a civil war over it. Those are all really useful things that autocracies have to cope without. It's not like it's easy mode.

It's not that security is fake, it's that giving up liberty doesn't naturally produce more security, and pursuing greater liberty doesn't necessarily erode security either.

It's not like pre-Revolutionary America was a notably secure place that inevitably see-sawed into a freer but insecure place afterwards.

It used to require some real elbow grease to write blogspam, now it's much easier.

I hardly ever go through a post fisking it for AI tells, they leap out at me now whether I want them to or not. As the density of them increases my odds of closing the tab approach one.

It's not a pleasant time to read Show HNs but it just seems to be what's happening now.

It is at least thematically appropriate, of course a corporate-built language machine speaks like LinkedIn.

The actual mechanism, I have no clue.

Well, quite. And in an American Revolution context it's not like the colonies were notably less secure places to live after they gained independence.

I think the security/liberty tradeoff is actually often a false promise. You can end up trading away liberty for nothing at all. I don't like buying into this, even to say "liberty is better, we should do that instead" because it implicitly concedes that you would really get the security on the other side of the bargain.

And if you don't get the security you were promised, it's too late to do anything about it.

Was there much with better effects on TV at the time?