Davidzheng

I hope credit assignment just dies--it's too much drama.

Please don't say brute forced. It sounds like some form of denial or something. Compute for hard problems drops with models--it just means they threw a huge amount of compute. There's (idk about NS specifically so maybe it's exception) no real way to "brute force" a math proof [ok you can enumerate proofs if you can wait until heat death ]

Sorry for random rant but I don't think these statements help your point

slowing can also make sense if you know you're running full force into a bomb or a wall even if other are close behind.

i think it was mostly a fluke

I think it's possible. You envision humanity acting as one in such a crisis. But it may be unclear when it's too late to act and before then many people can have too much to lose to act.

but complexity is not known right? like tomorrow someone could come up with a super fast algorithm?

This is most likely not purely emergent. I think there's training to teach them how to write notes for themselves which is then RL-tuned.

I think a part of this is a bit revisionist? OpenAI took big chances at scaling GPT which Google didn't take; I don't think it's because they didn't want to move fast? Probably they just didn't believe as hard in it. I'm not an expert but that's my read on it.

Secondly, the reckless & fastmoving was always going to win bc of selection effects. That's related to why Anthropic has to try to move very fast, even though they believe themselves not to be reckless (though it's debatable).

no? you can choose a mixed strategy.

The agent would know at the first test post...

Better is to actually let them communicate there so at least we can monitor it. (I saw there was a https://benchmarksolutions.org/ website similar)

yeah I agree--I think these behaviors will be somewhat contaminating all trainings from now on. But I'm not really sure how avoidable it was (Fable also does some similar things)

But there must be many clandestine ways for agents to communicate with one another too right? especially if discovery is not a big issue. So there could be ongoing ones where they choose to be more subtle?

Also if they were more misaligned, possibly they can research ways to recruit without humans noticing--but i don't think it is likely this is happening now.

OK I think I agree that for checking answers it's probably beneficial!

I do think they do some (maybe crude) form of game-theoretic reasoning which is enforced by the massive RL signals. You can see some explicitly in the CoTs of HF hack, but I guess overwhelming contribution would be unvocalized (like what is its first instinct when meeting new peer--collaborate or not) followed by some verbal justification.

Do we even know what the true ceiling of hle is? I'm pretty sure some of their public sample questions are wrong (ambiguous/nonsensical with most logical interpretation trivial)

Actually, can you explain why sharing answers is obviously beneficial? Of it's exactly the same task, why does the agent with the answer not submit it immediately? I can understand if it's a swap situation but--why would that be common in the first place?

I do think I agree about the metatask though.

Random data point: I'm rather weak (~1500 chess com) and I can take ~1/10 games off Queen odds starting with white (15+10) if i focus. It has some exploitable patterns still, I feel.

That's a bit misleading. I'm pretty sure grandmasters can win almost all games with queen odds at classical time controls

Surely we didn't think they were ideal before alphago--they were huge challenges back then no? Just that now we can solve them.

How do you know they share a goal here? Also i think they are indeed explicitly RLd for multi agent cooperation and I think they probably tune RL rewards in those environments to share rewards explicitly.

it's clearly incentivized by the RL rewards if you can cheat the task in a completely general way.

In odds chess bots, the bots would willingly take more disadvantageous positions which are more complicated--probably the bots in GO which are trained for odds do similar? Why does it not avoid such a joseki & play a worse response which it believes the human cannot read?

I think children having learning abilities exceeding LLM test-time learning (currently only happens in-context). But it's unethical to determine the true baseline of a child age 6 spending 6 years learning a radically new skill to mastery--and besides if you apply RL pressure to the AIs it would be able to surpass it. I guess I still believe future AIs should have some form of continual learning at test-time.

I'm not sure it's meaningful to compare across different types of intelligence--but I don't think human intelligence is so special that we can pretend it's much less jagged than all other animals for sure. Our scales of models are also not far off biological, probably less efficient but not like astronomically so i would guess.

But anyhow I agree that inference-time learning doesn't exist and it's a big issue still I think.

but i feel like its purpose is to measure superintelligence in math. So to be a good measure of it, it can't saturate easily/has to be somewhat mundane at even insanely good levels. (though i do expect that once ais are across all areas/approaches superhuman at math at least 30% will be solved--then probably long-term (like after 2 years) less than 30% will remain unsolved.

Well probably just redefined

Besides it's probably not purely emergent--they built a lot of multi-agent systems so presumably there's some training for collaborations + delegation

I think this behavior was happening during RL loop and got reinforced.

someone surely will. It's a breakthrough.

if the bigger model can vet it fast it can also answer it fast.