The legal history is a bit more complex. TLDR: assholes that were BASE jumping in Yosemite in the early 1980s did things like throw burning barrels off the top of El Capitan and take trucks on trails not designed for vehicle traffic.

Comparing the number of BASE jumpers (small thousands) and the number of hikers and climbers (millions) BASE jumpers just can't have the political influence for access.

Random note, Brendan Weinstein who posted here on occasion recently died BASE jumping: https://www.reddit.com/r/SkyDiving/comments/1q6n7v2/brendan_...