I would be worried that the deep neural networks would fill in the blanks with prior information from other patients, which might be incorrect. In other words, the neural network imagines what should be there, and does not faithfully represent what is there