No, it's both.
People correlate to things like schools. Buildings correlate to things like the fire department, precisely because -- as you put it -- an empty apartment building will catch fire too. And construction regulation, and the courts, and all sorts of stuff. There are a lot of government services that really are building-oriented as opposed to person-oriented.
IMHO schooling ought to be funded entirely at the state/federal level, not the local level, just for greater economic equality and opportunity. But since it is funded at the local level, and people don't like high sales taxes (and taxing visitors buying stuff to pay for local schools is weird), and local income taxes often don't exist, it ends up being property taxes. Still, there are some states that do have local income taxes, which is generally a much better solution.
If an arsonist sets fire to an apartment building this is a bigger problem (bigger fire, harder to fight) than if they set fire to a house.