The dermatologists make small dots/arrow markings on the positively identified lesions. What the Nature paper apparently didn't do was remove the small markings in their training. It is probably an innocuous flaw since gradient-based investigation/ interpretation only took off later than their publication (2017).

As a result, instead of positively identifying the lesion based on disease pathology, it identified overwhelmingly based on presence or absence of medical marks. There was following up discussion in a certain paper of this experimental design (I can't remember the exact name), but they did gradient based activation mapping and those pointed to the marks as the identifying feature. It felt quite a revelation of why this worked so well.

More information:

1.https://jamanetwork.com/journals/jamadermatology/fullarticle...

2.Swetter (2020) .Novel Technologies to Improve Melanoma Detection and Care Focusing on Artificial Intelligence

3.ISIC Workshop 2019 at CVPR (Slides online)

4.https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8074854/