A thing is worth what another is willing to pay for it.
The problem with land, unlike almost anything else, is that you can't move it (or trade for a fungible equivalent). People pay high prices for urban land because that's their preferred way to get closer to other land they want to be close to. When it's important enough to you to have that lot, you are willing to trade a lot of value for it - which may include liquidating some of the value of what you intend to put on the lot (million-dollar lot, build cheap bungalow), or working with others to figure out how to put a lot of inexpensive housing on it (billion-dollar building with 1000 million-dollar flats).
The bankers just help you figure out how you can rearrange your assets & income to free up that million dollars now - when you're willing to pay more later to have the money now.
And nobody is forcing you to do anything. It's your choice to decide whether having that land, right there, is sufficiently valuable compared to the value of what you intend to do with it.
Most land is cheap. It's the land you want that's expensive - usually because a lot of other people want it too, and you're all bidding for it. No imposition of "fair" is going to be any more fair than supply-and-demand.
Want cheap land? Go to zillow.com, enter your state as a search area, and set the max price to whatever absurdly low value you want. I've found many lots for $1000, $100, and even $1 (including a lot in Atlanta for $1; no, they're not going to show you what's inside the building, take it as-is).