That is because they are the prettiest and most desirable destinations. This is unsurprising. People want to see Yosemite, not some barren nothingness out near Ridgecrest, CA or Kingman, AZ.
Just because land is owned by the USFS or BLM and has public access doesn't mean it has any real recreational or attractive value that would draw visitors.