* Massive class imbalance -- we have much more data for healthy patients than for sick patients, especially at the scales required for deep learning.
* Heavy amounts of noise -- Medical imaging is a difficult feat and noise is a fact of life. Not only that, but data is hard to come by and some noise in the labels is likely.
* Long term outcomes -- trying to predict diseases or other long term results from signals is very difficult, especially when the outcome is not immediately obvious by looking at the input.
TLDR: Imagine that you have to compete in ImageNet, but the images are full of noise, half of the ones labeled "dog" are actually some sort of bird, and instead of guessing what's in the image, you're given an image and then asked to guess if that image, when painted in watercolor, will make a baby smile.