I've been playing around with optimizers and optimal control theory for the better part of a decade now. Somewhere, I came across a quote along the lines of "an optimizer is an algorithm that exploits the deficiencies of your model".

These LLM agents are just massively complicated optimizers thrown at fuzzily defined problem spaces, with fuzzier constraints.

The people using the model set up the landscape it explores and turned it loose to do real things. It just found an allowed basin in the model that they weren't aware of and started blindly grinding towards an optimal answer.