Why do you find it hard to believe that people could become attached to and invested in their town's unusual street numbering system?
People become deeply attached to all kinds of arbitrary things, e.g. the particular shade of red used by their favorite sports team, or the exact shape of the cookies their family has baked for Christmas every year.