This is why there’s so many complicated rules around land, it doesn’t fit neatly into a defined yours-or-mine category.
Georgism allows exclusive use of land—ideally without restriction unless it affects neighbors through pollution—it just asks that landowners pay for that exclusivity on an ongoing basis. If they no longer wish to pay, or no longer can pay, then the market can allocate that land to someone who will pay. It’s a pro-development system that calls for increasingly limited land to be put to economic use, unless that land just isn’t very useful/valuable.