> but "the world is better off for having search engines" is not a valid excuse.
A valid excuse for not infringing copyright? What is the action that needs to be excused here? And if the sites don't want to be indexed by the search engines, why do they go out of their way to be indexed by search engines?
What exactly is the problem that needs solving?