How to measure performance in machine learning

This is a great example on how one can get fooled by optimizing the wrong measure. This should be introductory material to machine learning, before any algorithm gets introduced.