> extrapolate the popular vote over the rest of the non voting population isnt fair

I don't know what you mean by "fair", but the non-voting population has much less influence over politics than the ones who vote. Regardless, we have surveys showing the opinions of non-voters and Trump, at least a couple weeks ago, was at record levels of unpopularity among Presidents-elect.

But I don't see how it's relevant to the point, which is that elected officials are compelled to and do respect the opinions of voters.