One of my friends went down a path of using ChatGPT to validate his belief that his alternative medicine approach was better than what his doctors recommended.

He had a whole host of tricks to work around ChatGPT’s protections or cautious replies. He’d strip out the cautions because he thought it was just OpenAI’s lawyers forcing disclaimers into the model. If he didn’t get the answer he wanted, he’d just retry or rephrase until he did.

I think the other half of the problem is that LLM users can be very good at pushing the LLM into doing confirmation bias. It’s much easier when you can hit the retry button with no side effects, unlike a human who will recognize what you’re trying to do when you keep asking different variations of a question until you get the answer you want.