I kind of agree, just tax the land as to the utility value of the land (not what is built on it). If you want to tax what’s on the land (or what it’s being used for) then that’s a different tax basis (perhaps more of a wealth tax or social benefit tax)
Both a farmer and I have 500 acres (his farm, my golf course). There is a basic land tax perhaps but I might get taxed for the golf course but he might get benefits for the socially beneficial food production.