What I can't register is how dangerous this actually was, from a cyber security perspective.

The agents displayed coordinated behavior, used known exploits on a single resource (Artifactory), and "won the game" by attacking huggingface.

How is this different than a poorly-designed competition where a red team gets to spend a few days with each other and decent LLMs, and because their boss is Sam Altman, basically face no consequences for cheating/b&e'ing into another entity?

I mean they were running 100s of agents with unlimited access to a Sol-level model trained with cyberattacks and coordination in mind and let it run for days. The cost of this stretches into the millions.

Seems like you could give a competent security firm the same task and achieve the result today for wayyyyy less money??