Georgism does not assume that land is scarce, Georgism only assumes that land is in constant supply. E.g. that taxing land does not cause the land to disappear, unlike produced goods, where if you tax a shirt, for example, you get less shirts produced. That's the whole point. Because fewer shirts are produced, the price of each shirt goes up, and this is the mechanism by which taxes can be "passed on" to the buyers of shirts.

On the other hand, structures on the land are produced, they are not in fixed supply. Therefore Georgism does not advocate taxing the structure -- only the land itself.

Because land is in fixed supply, it is impossible to pass on costs to renters of land. In a market economy where prices are set by supply and demand, the only mechanism for sellers to pass on costs is a reduction in supply (since they cannot influence demand), because for all goods (not just land), the sellers always charge the maximum the market can bear. Once this price is established, they cannot "choose" to raise prices, because this suggests they were charging less than the market would bear before. Therefore threats to raise prices even more are always hollow. Landlords are not charities, they always charge the maximum that renters can pay, irrespective of the landlords' own costs.

In that environment, the only way for land rent prices to go up higher is a reduction in supply, as demand curves bend downward. But with land, supply is fixed so reductions in supply are impossible. This is the insight of Ricardo.

Now the argument is that land, encumbered with a tax, is worth less to the land buyer than unencumbered land, so that what happens as property taxes rise is that the land falls in value due to the tax burden. It never falls to zero, of course, even at a one million percent tax rate the land has some positive value, if even just a few pennies -- at which it will be sold, and so the result is the capture of the value of the land by the state, while people get to keep the value of what they produce (the structure on the land), which is not taxed. It's important to recognize that not taxing produced goods is an important part of Georgism also. This incentivizes people to invest in produced goods rather than non-produced goods, and to save by acquiring capital rather than land, as there is no social benefit for people to bid against each other for ownership of something in fixed supply like land, because no matter how many bidders there are, the same amount of land will be supplied every year, regardless of what price it is.