I'm unclear what you consider reasonable due diligence in this case. Should the protagonist have assumed that the gym API was unauthenticated, and that the agent would discover this?
They should not have given a next token predictor unsupervised Internet access, regardless of what the marketing might have told them about its capabilities or alignment.
Don't use technology you do not understand in ways that might bring harm to others. Exercise appropriate caution.
I don't think the guy itself is fully to blame though. Our whole culture has rotted quite significantly.
Yes.
> regardless of what the marketing might have told them about its capabilities or alignment
I think that marketing teams should be blamed for lies and exaggerations much more then people who trusted them. As is now, everything is too much biased toward the biggest liar out there.
Might "harm innovation" though.
As programmers, we damn well have responsibility for that which we write and execute. AI cannot be treated any other way. And that doesn't mean cutting corporate structures a check to pressure engineers as accountability sinks either. There needs to be a formally recognizable standing for a practitioner to tell their employer "no" that is recognized and backed by the public. Otherwise degenerate/harmful use cases cannot be professionally prevented. There'll always be a more desperate developer willing to take the risk.