Sounds like a very reasonable thing to do unless the author explicitly asked it to not search the web.
That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results.
There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes.
Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems.
No thanks. I run all my agents with lots of permissions, but I don't want them to have access they have no need for any more than I want coworkers to have access they have no need for, because whether or not you can "get alignment", mistakes get made all the time.
I don't want to have access I have no need for, and frequently tell clients to take away access to things when I no longer need them for exactly that reason.
https://github.com/nburns/dotfiles/blob/main/AGENTS.md#tools
people would love it if LLMs were deterministic and never hallucinated. It's just that the technology to do so isn't possible, so we make do with fuzzy analog machines with digital controls because we don't have digital machines with digital controls.
The reason people prefer llms over programming is exactly because it lets them specify their program fuzzily, i.e. it lets them avoid going to the trouble of specifying enough detail to make it deterministic.