Places can become more or less desirable to live in - all this talk about matching inflation rates etc. is about the housing market as a whole - average. If you bought in a bad neighborhood and suddenly the neighborhood gets renovated and richer people start moving in - your property can increase in value multiple times over what you paid for it initially. If your area has excellent job offers and limited housing supply you will see housing prices increase because more people with more income will come to that area to work there. The problem happens when the local job market hits a slump and you assumed the demand increase will fuel the growth to cover your investments.
Of course, others I know have also lost substantial value at times.