And I'm not saying NP + ChatGPT - it should be properly calibrated system which would defer to a 'proper doctor' in more complex cases.
If the prompt is an expert-written board question! Not so with inferior prompts [0]. Critically, you need deep medical knowledge to interact correctly with the agent.
What you're asking is basically: "If we take someone out of a three month dev bootcamp, and have them prompt Claude, why can't they be as good as a four year CS grad?"
I doubt that you would feel similarly about expertise in your own field.
I'd wager GPT-6 would not depend on high-quality prompts, although it might still be good to get a trained person to enter information and do a sanity check.
But the core problem still exists based on the clarity of the prompt and the prompter.
Seeing my wife's depth of intuition and expertise as a physician, compared to my comparatively caveman-like prompts when asking for health advice, there's a massive delta in accuracy.
Have you ever had a significant health issue and tried to get help from a model? Even something like GPT 6 Pro? The firehose and confusion you will generate trying to figure out answers are far less helpful than having directed interaction with a physician.
The issue is that the AI is at best what's in papers and medical records, which often forgoes the core thing that might lead a physician to uncover something or take a different approach with the patient.