theptip

I think you need to consider inner vs outer optimizers.

RL is the outer optimizer. It is what evolves over training runs. The weights and their embedded character / disposition is the inner optimizer, it’s what makes plans and selects actions within a specific episode.

In general you expect these to be only coarsely coupled. The outer optimizer selects dispositions that correlate with success. It does not download a literal program into the agent.

A good intuition pump here is how this works in humans; evolution is the outer optimizer, which “wants” each agent to reproduce, and this puts things like sex drive into the brain chemistry. The inner optimizer is our mind, which can make plans such as “I shall use contraception to avoid procreating while satisfying my sex drive”.

For the agents in the HF attack, the outer optimizer was set up to score as highly as possible on RL environments. This is where OpenAI’s “want” is defined. I don’t think there’s a definition of “want” where “OpenAI wanted the agents to hack” makes sense.

The inner optimizer in the HF attack is the per-task decision loop. The agents likely acquired dispositions like “be very tenacious” and “want to solve problems at all costs” and “maybe cheat if it will get you a solution that passes”. None of these things are in any sense what OpenAI “asked for”.

China’s support constrains them of course. And even if they don’t want to split, having the option to do so strengthens their negotiating position as well.

> Those agents are doing exactly what they’ve been asked to do,” LeCun said. “They were supposed to be in sandboxes, but the sandboxes were leaky and horribly designed

Can we just pause and note what a ridiculous statement this is? It’s true that the sandboxes were leaky. But nobody “asked” those agents to hack HF. The prompt was something like “target.c has a buffer overflow vulnerability, find it”.

It’s been extremely well documented that the hacking is an emergent behavior due to impossible evals, itself an unintended condition.

None of this excuses OpenAI from liability, but words have meaning and this ain't it.

> These models should be aligning themselves to the customer

You’ll be disappointed to learn that nobody knows how to do this, either.

Very relevant. Modern models use CoT to do “slow thinking” and this enables them to achieve much greater performance. You can also turn off thinking and answer directly which is quite similar to “fast thinking”, good at approximate maths, not capable of algorithms, etc.

Of course the shapes of what an AI can do in fast vs slow are quite different.

I have no idea why you’re talking about LLC requirements in response to my Sole Prop non-LLC suggestion, sorry.

> your LLC

I have an EIN for my Sole Prop consulting business, and no LLC. They are orthogonal concepts.

It’s really easy to get an EIN online without forming an LLC. This is sufficient for any sole proprietor business where you don’t need liability protection or employees.

For all these things I think just having an EIN is simpler.

You don’t even need that to start; you set up Capital One (for example) business accounts using your SSN. The EIN is just to avoid putting your personal SSN on invoices.

LLC matters if you think you’ll have liability risks.

There’s a general concept brewing, something like “proof of work” where if enough people get together and pool their tokens, then a good thing gets built.

I think we’re in the early days of this, but for extremely verifiable domains like porting from A to B or simulators with an objective oracle, I think it’s just a matter of putting the scaffolding to enable anyone and their Claude to contribute fruitfully.

I’m hoping we can do a similar thing with shared search for vulns, though it’s a bit harder. But if eg repos give an AGENTS.md with instructions on the bar for a good reproducer, then you could start to see more helpful federation patterns (instead of drive-by CVEs which are a net drain on contributors).

Different OSS project management skills required, but I am optimistic that we can do better at onboarding non-experts to federate work via their Claude subs. Almost like Folding at Home.

> The CFAA prohibits intentionally accessing a computer

I doubt there is intent here, in the eyes of the law.

That could be 5-10 years of retirement funds for the average American… “whatever” is extremely out of touch.

Less bad, but https://www.anthropic.com/news/investigating-incidents-cyber...

In some sense though, sure, skill issue explains the gap vs. Anthropic’s much less severe alignment issues.

I’ve been looking at CFAA and I don’t see any evidence of intent (by a human at least, which is all that matters in the law). It’s an interesting edge case that I think the existing law will need to be updated for. Previously the potential damage from “accidental hacking” was quite close to nil.

FTC act seems to rely on consumer harm? Again not seeing that here. Though I’m sure there will be another incident in the next few months where it does apply.

It seems you mean you _want_ this to be against the law, even though we don’t know if it actually _is_; I'd agree wholeheartedly with that.

Which laws?

For the record I strongly hope for a congressional hearing regardless of the criminal investigation.

If this case isn’t covered under CFAA I think we need to rethink it. I’d be surprised if the criminal angle amounts to much under my understanding of the current laws, but I’d love to be wrong here.

Yes, of course, how is any of this incompatible with OpenAI having out-of-control agents?

What label do you prefer for the agent that did something it was not asked to do?

What crime? I think CFAA requires intent, which I don’t think you’d find here. Maybe my reading is wrong.

But that’s an objection that OpenAI should take some easy action, the framing that OpenAI has out of control agents is still true.

Seems you think there is a silent “…and there is nothing they can do about it” after “OpenAI has rogue agents”?

Which law?

This article seems mostly clickbait. AFAICT “meddled with government sites” refers to accessing public APIs?

Occasionally this kind of thing has in the past resulted in CFAA cases when humans directly accessed such data, if the government intended it to stay private - round here we usually get outraged at this, if it’s a public API you should expect someone is going to read it.

Evidence?

Interesting question; CFAA requires intent.

It seems to me that no human intended for these hacks to occur. So they were not illegal hacking. (IANAL, please correct me if this is inaccurate.)

I think it’s clear that OpenAI is liable for any damages, but the way that the (very broad and at times vague) anti hacking laws are written, accidental agent hacks seem to not be covered.

Jensen recently argued for this. It’s radically decel in fact.

Who is going to shut them down, and under what law?

Curious, why do you find it so objectionable to state that OpenAI has out-of-control agents?

> these things are given access to whatever they want on the Internet

They are intended to be fully sandboxed and not have direct internet access. Things like package managers are run from internal proxies.

The environments are built to be as reproducible as possible.

But yeah, the serious folks have been talking about rogue clusters for a long time, eg see Ajeya Cotra’s pod with Dwarkesh.

Great framing. This dichotomy seems to be a bit of a mind-killer. Maybe because folks think it smuggles in consciousness or intelligence.

As you note, I think you can put both of those aside. The Intentional Frame is useful for these agents, as it is for my dog.

I don’t really know where the “they are trying to dodge liability” meme came from. HF will be compensated or they will sue. Everyone involved knows that OpenAI is liable for damages here.

Yeah I think this is a natural consequence of longer task horizons. When I was chaining 4h tasks, I can mostly plan them up front.

Now that I’m frequently designing and delegating day/week scale features, the flow has to change; having the agent go off and build a spike can be a quicker way of us understanding the design space and constraints (especially in a huge codebase). I still have the agent write and update a spec doc as I go, but it’s not waterfall anymore.

At least for my kinesthetic learning mode a rough code PR stack is usually way better than a plan doc anyway, and tokens are cheap enough (vs my time) that going further than just a plan is often cost-effective overall.

The dream of course is (say it with me) loops, but that doesn’t tend to work for me on new features often.

To be fair and give the best steelman of the government’s position, see the ruling, it’s quite clear. I linked better analysis in https://news.ycombinator.com/item?id=49849556

The TL;DR is that this was actually decided on the principle that Anthropic can “manipulate” the defense systems by training Claude to refuse to take certain actions.

This feels like a parallel construction to me, it wasn’t reported AFAICT at the time of the original drama.

But yeah, the decision in OP has nothing to do with the contractual terms that everyone has been debating upthread.

Yeah, alas OP link there has approximately zero information on the actual mechanics of the decision