Neither agricultural nor wilderness trust land are counted as vacant. There's specific zoning and tax rules in most municipalities for them.
Land that is zoned for some form of residential, commercial or industrial use but doesn't have building is considered to be vacant.
The US doesn't have a shortage of land. However, vacant properties (residential or commercial buildings with no tenants) are indeed a problem in densely populated regions.
For some areas, it's because the city population shrunk and nobody wants to live there (Detroit comes to mind as a semi-recent example) but for others it is speculative investors who don't want to be a landlord and are just betting the price will go up.