Very good points. The Normal Distribution has very nice properties, which adds to its popularity, which, if I understand you correctly, adds to the significance we place on mean and variance.
There's another interpretation of the mean (and conditional expectation) as quantities minimizing squared error. It's not surprising that squared error and variance are so similar and that these are connected.
CDFs and characteristic functions determine distributions.
This may be pedantic, but an object that determines a distribution that isn't based on expected values is the CDF. In my first probability course in undergrad I think we defined probability mass functions and probability density functions first and defined CDFs in terms of them, but from a measure theoretic point of view, the CDF is more fundamental since it is defined for continuous and discrete distributions (and also for distributions that are neither).
Of course the expectation value and the probability measure are really two sides of the same coin, the expectation value being equivalent to integration with respect to the probability measure.