This is a limitation that I am encountering more and more when casually talking with ChatGPT (it probably happens with Claude as well) where I need to prompt it with as little bias as possible to avoid leading it towards the answer I want and instead get the right answer.

If you open with questions that beg a specific answer often it will just give it to your regardless of it being wrong.

Recently:

"Can I use vinegar to drop the pH of my hydroponic solution" => "Yes but phosphoric acid [...] should be preferred".

"I only have vinegar on hand" => "Then vinegar is ok" [paraphrasing]

Except vinegar is not ok, it buffers very badly and nearly killed my plants.

"Should I take Magnesium supplement?" => Yes

"Should I take fish oil?" => Yes

"I think I have shin splints what are some ways that I can recover faster. I have new shoes that I want to try" => Tells me it's ok to go run.

An MD friend of mine was also saying that ChatGPT diagnosis are a plague, with ~18-30 y/o people coming and going to her office citing diseases that no-one gets before their sixties because ChatGPT "confirmed their symptoms match".

It's like having a friend that is very knowledgeable but also an extreme people pleaser. I wish there was a way to give it a more adversarial persona.