If you ever go through the process of buying vacant land you’ll find the biggest risk is whether the city will ever let you build anything on it. Even a house being there once before isn’t a guarantee. It could be the green belt, too close to an airport, or just zoned wrong.
At least in Canada land is not that scarce (even in desirable areas like the GTA) you just aren’t allowed to build on most of it.
This is why land development is so commonly associated with corruption. The best way to make money is to buy up artificially restricted land and bribe the government into letting you build.