I mean the thing about fungi is that the tops can look the same and you just need to do a spore print to positivity distinguish one from another. There may be some mushrooms which are simply impossible to tell apart by outward appearance. In one of the fast.ai lectures Jeremy shows how to distinguish different breeds of cats. Then he shows how to look at the confusion matrix, and he found one pair of breeds the network really struggled with. It turns out they look really similar to him too, and when he researched further he found they’re simply hard to tell apart. Perhaps with an enormous data set there might be small differences a network could detect, but the confidence might still be low.

And given that mushrooms can kill you, it may simply never be advisable to rely on any photo based identification.