Isn't this a bit strong? E.g. just because it is not a good idea for your use case right now, it may not mean that in the future something else happens and it could invalidate your firm proposal.
Isn't this a bit strong? E.g. just because it is not a good idea for your use case right now, it may not mean that in the future something else happens and it could invalidate your firm proposal.
The whole craze about my application i.e. dermatology successes in ML spurred from the 2017 Nature paper, where skin cancer was detected as good or better than dermatologists. But technically, such experiments have design problems: We had apriori knowledge of the dataset (White N American Melanoma data) & hence we could ascertain the model performance. Real world data is much more variable. Further, later it was revealed that ML model latched on to the little marking physicians made rather than generalizing on lesions. 3 years later my experiments could model reliably only on 10 very common diseases & of a very uniform skin type and ethnicity. Those results were nowhere close to perfect.
The proposal to keep Human-in-the-loop is a much fairer alternative in ML aided medicine, than end to end machine learning. Most direction of research is headed that way. Physician assistance is much more reliable than potential replacement.
The trouble with synthetic data is that it doesn't address the extended variability in real population & generalization will always suffer. Also, doctors take multi-path decision, choice by elimination, past cases - based on several diagnostic inputs & even gut intuition. At that scale of input multimodality, Type I & II errors are at a scale higher than correct identifications. And we don't know how to teach intuition or imagination to machine models well enough. Those fall back to rule based methods & edge cases.
... excuse me?
As a result, instead of positively identifying the lesion based on disease pathology, it identified overwhelmingly based on presence or absence of medical marks. There was following up discussion in a certain paper of this experimental design (I can't remember the exact name), but they did gradient based activation mapping and those pointed to the marks as the identifying feature. It felt quite a revelation of why this worked so well.
More information:
1.https://jamanetwork.com/journals/jamadermatology/fullarticle...
2.Swetter (2020) .Novel Technologies to Improve Melanoma Detection and Care Focusing on Artificial Intelligence
3.ISIC Workshop 2019 at CVPR (Slides online)