Bolts have always been a source of controversy in climbing – in fact, bolts were the main controversy in the first ascent of the Dawn Wall (/ Wall of Early Morning Light) back in 1970, so much that the first ascensionist's rival Royal Robbins actually climbed the route and chopped the bolts off as he went. There have been similar controversies all over the world, another notable example being on Cerro Torre in Patagonia.
Unless a climbing area is a designated "sport climbing area", bolts are generally not placed unless there is no other way to protect a stretch of climbing. Climbers try to place devices that don't damage the rock (nuts, cams, tricams, hexes, etc.) first, but sometimes none of those devices will work as they all rely on jamming against something, and blank faces do not have the cracks and holes into which jamming gear can be inserted. When this happens, bolts are placed because there isn't any other way to protect the climbers in the event of the fall.
So, I don't really have a good answer to your question except to say that the use of bolts has always and still is a huge source of controversy in the climbing community. These days, with sport "crags" being the exception, climbers try to rely on other "clean" methods of protection before turning to bolts, which is in stark contrast to past eras where bolts and pitons were the predominant way to protect pitches.
The seminal article on "clean climbing" was authored by Yvon Chouinard and Tom Frost and appeared in the 1972 Chouinard catalog [2]
[1] http://en.m.wikipedia.org/wiki/Royal_Robbins#Wall_of_Early_M...
Which, it should be noted, describes a lot of terrain in Yosemite. The glaciated black granite provides limited opportunity for protection.
Climbers in general care a lot about the places they climb, and often tend to them as well or better than many other groups. There are organized cleanups and trail management and self-policing, and relations between land managers and climbers are usually OK, with climbers willing to cooperate with access restrictions for the benefit of an area (with the notable exception of Hueco Tanks and probably a couple other places).
If someone unnecessarily bolts a route, they can usually expect to hear about it from other climbers, and the bolts might even be removed. Given that casual observers often don't even realize the bolts are there, climbers really do fuss a lot over relatively little impact.
Of course, like any other group, climbing has some jerks too. (But proportionately less than many other groups.)
And before anyone screams, the debate on "fixed protection" is something best left to experienced local climbers. Every climbing area has their own traditions on the subject. The first rule is most always: Don't change the nature of the route to suit your climbing style. Respect those who setup the route before you. Putting new metal on an existing route, camo or not, is akin to spray-painting a Rembrandt.
Replacement of old/weak hardware is a different matter. Absent immediate safety, this should still be left to experienced locals.
Bolts allow cliffs to be used for recreational climbing, much like how trails alter the landscape, but allow the land to be used for recreational hiking. No human activity is 100% impact free, unfortunately.
Second is the fact that sometimes a bolt is actually the lesser impact. Bolts last a pretty long time, and if a route is highly traveled, the bolt can help offload what would have been gradual damage to the rock, flora, and fauna.
For example, if you have six climbing parties a day rappelling off a tree at the end of the climb, that tree is going to sustain damage over time. A bolt placed nearby can serve the same duty easier, safer, and with less impact to the tree.
To draw on other recreation, have you noticed stairs on hiking trails? A popular trail can experience so much erosion on steep sections that something so blatantly "leaving a trace" as installing stairs is actually the lower impact choice.
Generally the decision to bolt is favorable only when it's unsafe to climb with traditional gear. The impact that bolting has is rationalized worthwhile due to it's (arguably) low direct impact on the environment and the fact that sport climbing proliferates lots of outdoor enthusiasts with the attention and dollars that brings.
The general idea is that at some point in the future, we might be able to protect these difficult routes. So we preserve it for that. This argument is more legitimate in some cases than others. For instance, maybe someday we'll have some non-destructive way to protect thin cracks and flakes. But it is very unlikely that we can say the same about a barren face.
The concept of "fair means" is based on the idea of setting limits on the artificial aids that a climber uses to reach the top. For example, pretty much every climber agrees that taking a helicopter to the top is not fair means.
It's context-specific, too. On El Cap, taking the hiking trail up the back to reach the top is not considered a "climb" on par with ascending the cliff. But on Mount Rainier, hiking the top counts as a climb even if you take the easiest way up (basically because the easiest way up is still not easy).
Free climbing itself is actually a form of fair means--ascending the rock using only what the rock itself gives you for progress. All gear does is provide a safety net.
Climbing without bolts and pitons--known as climbing "clean"--is also a form of fair means. It takes as its guiding principle to leave the rock as you found it. Under this approach, climbers will indeed leave blank faces unclimbed. Those bold enough to climb such faces with minimal protection are respected as having shown great commitment to their ideals.
Don't you think there's any worth in preserving the natural environment for its own sake? It sounds like you're saying if you can't put bolts into the rock to climb it then the rock is worthless and shouldn't be preserved anyway.
Maybe if you can't climb something without permanently defacing it then you should just not climb it.