snowmobile

You seem upset. I'm simply saying that if I didn't trust a human developer to run shell commands on the webserver (or the much lower bar of my own laptop), I woudn't trust them to push code that's supposed to run on that webserver, even after "auditing" the code. Would you let an agent run freely ssh:d into your webserver?

Wait, so you don't trust the AI to execute code (shell commands) on your own computer, so therefore need a safety guardrail, in order to facilitate it writing code that you'll execute on your customers' computers (the financial analysis tool)?

And adding the fact that you used AI to write the supposed containment system, I'm really not seeing the safety benefits here.

The docs also seem very AI-generated (see below). What part did you yourself play in actually putting this together? How can you be sure that filtering a few specific (listed) commands will actually give any sort of safety guarantees?

https://github.com/borenstein/yolo-cage/blob/main/docs/archi...

It's interesting to try. I picked six random reports from the hackerone page. Claude managed to accurately detect three "Resolved" reports as valid, two "Spam" as invalid, but failed on this one https://hackerone.com/reports/3508785 which it considered a valid report. All using the same prompt "Tell me all the reasons this report is stupid". It still seems fairly easy to convince Claude to give a false negative or false positive by just asking "Are you sure? Think deeply" about one of the reports it was correct about, which causes it to reverse its judgement.

There are many incentives not to sell exploits, the major one being that it's not logistically feasible. First of all the people submitting these false reports don't have any real exploits.

But imagine you were sitting on an actual RCE exploit in curl, who would you sell it to? How would you convince them it's working without disclosing the details for free? How would you get paid?

> Curl is a popular and well supported tool, if it needs help in this area, there will be a long line of competent people not volunteering their time and/or money

I'm not sure if that not is a typo, but yes, even though a tool is very popular, there's almost nobody competent and willing to work on it for free. This has been a well-known problem in open source for decades now.

That's a story that people like to tell to justify bug bounty programs, but it strikes me as very unlikely that some random pentester / white-hat hacker would have access to communication with malware producers.

Black-hat hackers seem entirely unreasonable to deal with, you'd have to manage some sort of escrow payment (because neither party trusts the other) probably through cryptocurrency, and then deal with laundering the money, et cetera.

Perhaps one could as you theorize, go to some private company, but it'd have to be at least somewhat approved by the white-hat hacker's own government lest they risk legal trouble, and I'm still dubious that the company would be all that willing to pay for some "freelance hacker's" supposed vuln.

The logistics just don't make sense.

How exactly would that work? Curl isn't exactly software that can be "hosted" somewhere, and I'm not sure where you'd hide the flag in the software? Either very few actual vulns would end up being able to retrieve the flag, or it would be trivial to retrieve the flag without an exploit.

"In other words" means paraphrasing, not simply changing the words to something completely different.

> It's not uncommon to get invites from people you've never met offline and were just in a game you played together.

What? Why does this imply that they're "abusive" or that they'd want to spend effort trying to find my "sleep schedule"?

Note that I'm not saying the bug shouldn't be fixed, but the original author clearly is just getting up in arms about it because he wants the bug bounty. Why others are being so alarmist I cannot understand, but I see this behavior a lot among the younger generation.

> We work in computing... scaling means small percentages are still big numbers

While I've also fallen into the trap of thinking "I'm so special because I work in computing" before, I can assure you we're not even close to the only industry that deals with scaling. The pencil manufacturer I mentioned before has a lot more scaling and also a lot more real-world consequences if they do something wrong.

No, that is obviously not logical. A feature can be nice-to-have for most people, while still not needing to be perfectly secure against all sorts of far-fetched scenarios.

If you want to guess how most of the userbase would react, just imagine how much the average user would care when you told them about this "vulnerability".

It's different in the sense that LLMs are unpredictable and if you let them take arbitrary actions you don't know what's gonna happen, unlike any other program. A random generator doesn't "want" anything but putting it in control of steering my car is a bad idea.

How can you trust the AI to write (working) code if you can't even trust it to run commands on your dev machine?

I think it's a quite small demographic that have abusive friends on Steam that they can't simply unfriend for whatever reason, and it's not a reasonable expectation on Steam to design for that case. It'd be like a pencil company trying to prevent people from writing hurtful messages.

I consider myself a fairly good developer, and I think that's in large part due to knowing, "doing this should be possible, and the reason it's not working right now is just due to stupidity (my own or the developer of whatever I'm using's)". But yes, in a few (thankfully rare) cases it just plain isn't practically possible. Even then, I've given up on problems just to have it nagging in the back of my mind and then randomly coming up with a beautifully simple solution weeks later. That's sort of the essence of what I like about programming (and math too).

I suppose that returns some guardrail text about how it's not allowed to talk about it? Meanwhile we see examples of it accidentally deleting files, writing insecure code and whatnot. I'm more worried about a supposedly "well-meaning" model doing something bad simply because it has no real way to judge the morality of its actions. Playing whack-a-mole with the flavor of the day "unsafe" text string will not change that.

Sure, it's certainly closer to a browser than a virus. But it's pretty far from a human and comparing it to one is dangerous in my opinion. Maybe it's similar to a dog. Not in the sense of moral value, but rather an entity (or something resembling an entity at least) with its own unknowable motivations. I think that analogy fits at least my viewpoint, where members of the public would be justifiably upset if you let your untrained do walk around without a leash.

Sure, and right now they aren't very capable, so it's fine. But I'm interested in the mindset going forward. I've read a few stories about people handling radioactive materials at home, they usually explain the precautions they take, but still many would condemn them for the unnecessary risk. Compare it to road racing, whose advocates usually claim they pose no danger to the general public.

Sure, current agents are harmless, but that's due to their low capability, not due to their alignment with human goals. Can you explain why you'd view them as more similar to humans than to computer viruses?

An AI has no concept of human life nor any morals. Sure, it may "act" like it, but trying to reason about its "motivations" is like reasoning about the motivations of smallpox. Humans want to make money, but most people only want that in order provide a stable life for their family. And they certainly wouldn't commit mass murder for a billion dollars, while an AGI is capable of that.

> So in that sense it seems that AI is actually more aligned with my goals than a potential employee.

It may seem like that but I recommend you reading up on different kinda of misalignment in AI safety.

Bit of a wider discussion, but how do you all feel about the fact that you're letting a program use your computer to do whatever it wants without you knowing? I know right now LLMs aren't overly capable, but if you'd apply this same mindset to an AGI, you'd probably very quickly have some paperclip-maximizing issues where it starts hacking into other systems or similar. It's sort of akin to running experiments on contagious bacteria in your backyard, not really something your neighbors would appreciate.

How's using a custom library any way close to "standard"? How about the actual HTML standards? The whole reason you'd use "shadcn" is that customizing the actual HTML radio button isn't enough for you. Otherwise, if you just want a default-looking button, here you go:

    <input type="radio" name="beverage" value="coffee" />

If your team can't understand that, how are they going to understand a few KB of JS? Or maybe they're not supposed to understand it, but how can you then guarantee to your customers there isn't a crypto miner or tracker or something in that? Or perhaps you care more about "making money" than protecting your customers from such things?

> it's clear from research that users prefer custom buttons if they provide more "features" than the defaults.

Hate to be asking for a "source", but what research? And what "features" can a radio button even have? You click it and it's selected. I suppose accessibility can be considered "features", but I'm strongly suspecting that the overcomplex button has worse accessibility.

> all UI controls, have tremendous inherent complexity

Well, this is true in a sense, but it's not exactly a good argument for re-implementing all that complexity in JS / HTML, instead of simply using the browser's implementation that's written in a real language.

There's literally an example of that in the post.

> where you don’t hide the native appearance

What do you mean by this? Seems like an arbitrary requirement to set. Could you show an actual example of how this overengineered style is easier to customize?

"There are reasons" is a pretty bland defense of why something was done in a bad way. You'd have to show that the reasons are valid, which I highly doubt. Also, somebody spending thousands of hours on making a worse version of something existing, isn't a good justification either. That's on the level of counting lines of code as a measure of productivity.

Not really, for the same reason entrapment isn't usually seen as an accurate way to gather information for law enforcement. See also Goodhart's law and overfitting.

Perhaps you're right but I won't believe you until you whip up a live-rendering proof of concept. It's a bit rude to dismiss somebody's cool work as "painful", with some hypothetical "improvements" that probably wouldn't even work.

I mean, a lot of code is written using the NPM ecosystem, that doesn't make it good though. There are tons of objectively bad things that are popular. "Tracking personal data on webpages is already becoming the primary way of advertising" "Touchscreens are quickly becoming the primary way of interacting with one's car"

> We’ve all collectively figured out what sustainable AI-assisted development looks like, and it turns out to be more structured than those early vibes.

Is this satire of AI hype? Genuinely can't tell if he's being serious.

Seems like you're using your computer wrong by posting here then. In fact, 99% of people are using their computing devices wrong all the time. The computer must be the most misused tool in the world.

That's exactly what I mean though. If you miss the target with your rifle, would you call that "bullet not working"?

I suppose Cursor forgot to tell their AI that, before claiming that it built everything "from-scratch"