The colosseum is 150 feet tall, which isn't exactly short. Beyond that point, I think a big problem is getting the resources up high enough to keep building, as well as convincing people to climb that far up.
@idlewords went there back in 2014, before the Saudis invaded: http://idlewords.com/2014/07/sana_a.htm
The first proto skyscrapers were built in the late 19th century. The Equitable Life Assurance Building (1870) and the Home Insurance Building (1885) were office buildings. The idea of a Penthouse appartment [3] dates from the 1920s. And as buildings are only slowly built and renovated, the first floors were still prestige floors well into the 20th century.
1 https://en.wikipedia.org/wiki/Piano_nobile
People don't realize what a paradigm shifting technology elevators were.
At ground floor (usually with higher celings than the rest of the house) dining room, sitting room, library, study, most of the "luxury" (marble, columns, paneling, paintings, etc.) was there, as they were the only parts of the house that visitors would see.
In the basement the kitchen and cellar (where cook and other servants worked).
On first floor (what probably our US friends will call second) the bedrooms for the owners.
On second (last) floor, right under the roof, usually with a reduced height AND extremey cold in winter and extremely hot in summer, the servant quarters.
Examples: http://0.tqn.com/d/ancienthistory/1/S/U/I/Rome_5.jpg
http://3.bp.blogspot.com/-z_W3YC_mG0Y/Tb55AUhJ0dI/AAAAAAAAAE...
The Yonging Pagoda was 138 metres high. It was destroyed in 534 when it was struck by lightning and caught fire.