This CloudFlare breach seems to have put a lot of people in a tough spot, but it feels like it's put archivists in an impossible position.
This CloudFlare breach seems to have put a lot of people in a tough spot, but it feels like it's put archivists in an impossible position.
First, we're talking about raw memory pages, not merely malformed HTML. Those memory pages might contain valid HTML, but most of the sensitive information is in the headers, not HTML markup. It won't be very difficult to write a script to identify documents where random headers and POST data have been inserted where they don't belong, or where the markup is so obviously invalid (even compared to similar documents from the same site) that there is a high probability of contamination. Having a full list of contaminated domains would obviously help a lot, because we'll only have to deal with thousands of domains instead of millions.
Second, contaminated documents by definition contain information that is NOT what the publisher intended to be crawled, indexed, or archived. So there should be less resistance to removing them.
Finally, most of the contaminated domains used features such as Scrape Shield that were intended to deter archival. It's as if the domain had a robots.txt that said "User-agent:* Disallow:/". I'm not sure whether it's even possible for the Internet Archive to archive such domains. If they can, maybe they've been doing it against the publisher's wishes. If they can't, well, there's no problem to begin with.
So I would completely disagree with your speculation about what's easy or hard. (Note that I've worked at a search engine and an archive.)
It might not be unreasonable to say that this privilege comes with a certain responsibility to ensure that those copies do not cause excessive harm to others.
So although Cloudflare is the one that fucked up, Google et al. also have a responsibility to do whatever they can to protect the public. They should do what they can, with or without Cloudflare's cooperation.
When there's an oil spill, we don't wait for the oil company to come and clean up their own mess. Others clean it up a.s.a.p. and (ideally) then make the oil company pay the fines and damages. CloudBleed is a virtual oil spill. They literally sprayed other people's private data all over the internet.