That's arguably for regulators to decide though, not technologists who have no exposure to the harm they cause. I can't even say "engineers" because engineers are held to a standard of care [1] that people slinging code are not. The fact that medical practitioners are sometimes wrong is not a legitimate means to wave away AI safety.
In this instance, the solution isn't immature tech (that may [may!] one day be good tech), it's policy that enables healthcare access. You aren't going to fix that with LLMs. But, you know, that's what tech people keep doing. Poor tech solutions to people problems, and sometimes getting wealthy due to luck along the way.
[1] https://docs.acec.org/pub/18803059-a2fd-2d06-cc39-a6d1dd5752...