I have stopped traveling to dive, but because it's too sad, not because I was concerned about destroying the ecosystem by viewing it. I don't touch the reef, I don't fin near coral, I can hold my buoyancy quite well... but the same can't be said for many divers.
That said, there is a factor for people that if something is out of sight, it's out of mind. Being involved with the ecosystem really is the only way I've seen protection of it happen, people just tend not to try to protect something they aren't engaged with. I'd rather everyone dove, even if that brought traffic to certain sites up, because I think they'd make the macro changes we need to make to preserve things while accepting the micro change in boat fuel use or reef traffic, or whatever risks you're thinking of. Seeing a fish trying to escape from a plastic bag is a sobering experience, you only need to see it once.