> I must believe there's any easy way to eliminate some "outliers" using mathematics, but I can't recall the function(s) to do so.

The median is one good way, as you already have. You can also use the interquartile mean: http://en.wikipedia.org/wiki/Interquartile_mean