It’s also come to mean the densest part of any city. Even the town I group up in would call the densest part “downtown”, even though the only buildings taller than two stories were city hall and the historic theater.
I always thought "downtown" and "uptown" were more like "south from X" and "north from X" where X is some reference point in context (e.g. "where I'm standing"). Turns out, it's more similar to "downwell" and "upwell", i.e. towards or away from some central point (gravity wells being spherical-ish and all).
It just so happens that for Manhattan which is the most famous uptown, all four of those indicators point in the same direction. In general I think an area would more likely be called an uptown if it met more of those criteria relative to some downtown. (And there would be correlation, wealthier residents would buy higher-elevation land.)