A few thoughts below as a medical student with some ML background. I'm quite interested in hearing what others think:
1. The majority of clinical ML applications are still focused on narrow (e.g. binary) image recognition tasks. This makes sense because it's where CNN's excel. Also, many of these tasks are well reimbursed. I expect the performance to continue improving as datasets and annotations improve.
2. There is less activity on broad image recognition tasks, like reading a CT Abdomen and Pelvis. Here the domain of possible diagnoses is rather large. For example, patients sometimes have unusual anatomy from prior surgeries. Or, a cancer may obliterate common anatomical landmarks. An experienced radiologist or surgeon can figure out this anatomy in a few minutes, whereas ML may need new approaches to get to this level.
3. The history and physical exam are clinician-generated, not just for dataset annotations but for every new patient. This is notable because the history and physical exam determine the majority of what a patient needs. It is not only a matter of knowing which questions to ask, it also requires follow-up questions and a degree of mental processing before findings are written down in a note. A lot of patients will not fully cooperate without patience and empathy. So "AI" replacing doctors actually trend perilously close to AGI. This is both interesting, and difficult. Of course, NLP of human-generated notes / unstructured data is a very interesting area, but it still requires a human in the loop.