A spike is not smooth (typically meaning continuous in the variable and its first derivative), which was one of the conditions.
Then take a Cauchy or a t-distribution. Basically anything with a longer tail than exp(x^2). The Gaussian summary will be misleading because of the tails.