>it's mostly people who own 1-5 apartments in the city they live in
Owning 1 apartment is completely different than owing 5 apartments in the same city. You can live in 1 apartment but you can't live in 5 at the same time. You're profiting off the other 4 ,so you should pay tax for it.