These same people seem awfully shy when I ask to go through their photos, just to see if they have any using drugs.

But that's exactly it: if you leave a gaping hole for the feds to get through I (in the sense of a hypothetical attacker) can go through it too, especially with how easily most federal secrets leak out and the fact that rekeying is essentially impossible for this use.