With statistics you want to generalize. Find the simplest model that catches/explains most variation. Leaving that one outlier in would result in a worse model.