At this rate many exams will just become oral exams :-)
The only thing that has changed is the speed of access. Before LLMs went mainstream, you could buy whatever book you wanted and read it. No one would stop you from it.
You still should have a professional look over the work and analyze that it is correct. The output is only as good as the input on both sides (both from the training data and the user's prompt)
I don't think we can/should do this on today's LLMs, but if we continue advancing in the same way, and as-good-as-human reliability is achieved, the intelligence of a doctor is in your pocket whenever you want it.
And just like you say you know addresses because you have an address book, you'll know medicine because you have it immediately on-tap. Instead of holding all of that in your own memory, instead of having to use your own critical thinking (or lack thereof), just offload it to the LLM in your pocket.
We do this all the time with tools. Who now knows how to cut down a tree but lives in a house made of milled trees? There are so many lost skills that we defer to either other people or machines and yet each individual lives with the benefit of all those skills.
Tools make cognitive bypasses for us to benefit from. When we can make intelligence a tool, I assume we can offload a lot of our intelligence, or at least acquire new intelligence we didn't have before.
WebMD is the same whoever looks at it. An LLM can adapt to your clarification questions and meet you on your comprehension level. So no, it's not as naive as you are insisting.
The skill today is the application of that knowledge. If an LLM can provide the data context, and the application advice and you perform what it says, congrats you now have a doctor's brain on tap for your own personal usage. The doctor has it in their head, you have it in a device. The net differences are immaterial IMO.
An LLM is the doctor in your pocket. It's yours to use, and whether it is in your head (like a doctor who had to take exams to prove they really had it in their head), or in your pocket makes no difference in your ability to achieve a task.
"Intelligence: the ability to acquire and apply knowledge and skills."
Well, if I can acquire knowledge from the LLM, and apply it using the LLM's instructions, I now have achieved intelligence without doing an exam.
Problem is, I can lose my LLM. A doctor could lose their mental faculties though.