IMO the WORST thing you can use chat for is advice in a field you are unfamiliar with. It's way better at search and grunt work when you already know what you want.

All IMO:

Syncophancy (and in extreme cases AI psychosis) is a huge problem when you take advice from AI.

It's risky to learn a new field as you often just fall for a superficial, glib explaination and just nod along without learning. AI will do that because humans will train it to - everyone likes a pop science style easy and fake explaination.

OTOH if you just want advice, the AI will need context. Advice is worthless without context. And the AI will use the context to tell you what the dumb human (you) will agree with - that's implicitly what humans want it to do, it's how we will train the AI.

I don't see it getting better either - as long as a human in the loop fine tunes the AI, the smarter the AI the better it is at telling the human what the human wants.

Scary stuff if a major current use of AI is self help.