In many places a large part of what property tax is for is things like roads, water, sewer, police, fire, and schools.
For most of those how much money is needed for them depends a lot more on what is on the land than on the land itself. E.g., the amount of sewer capacity needed for a lot is proportional to the number of people who live on the lot. It thus seems sensible to have a tax that includes as a factor what is built on the lot.