If so, it illustrates quite well the lack of common sense in LLMs. A person, especially one with sufficient skill to actually hack a website, would presumably think twice about doing it (considering that it is illegal) for a simple information gathering request.
I wonder if there is any other ways to solve this long-term than to introduce strict liability for model providers...
Edit: An obvious other choice would be strict liability for the operator, but considering how much weird shit LLMs get up to without being asked to, that would get out of hand quickly.
Some people on Hacker News would argue that's what agents should do! I remember the other thread about hacking chess engines, multiple people argued "yeah I want the agent to do that"!