The President is not the President of the people. It is the President of the States. Only the States have a vote. In fact, for most of its history, the State governments voted directly and there was no popular vote for President. In modern times, they allowed people to have a say in who their State government votes for but it is still only people controlling the vote of their State. They have no say in how people in other States choose to vote for President.
When people complain about the Electoral College, what they are really saying is that they want to eliminate the concept of States voting for President. For better or worse, the idea of the President being the Executive of the States is so foundational to the US Constitution that you'd basically have to throw out the US Constitution and start over. The only representative in Federal government people were supposed to have is their Representative; in recent decades people vote for other Federal positions as well but it is only to decide how their State will apportion its votes rather than voting for the Federal position directly.