It looks like your neural network is overfitted on seeing overfitt where is none.

Prejudices is a form of overfitting IMHO