Bazel does not really hash any of the system stuff -- like system headers, and system-provided .a and .so files (unless they are explicitly declared in WORKSPACE, but this is optional, and I have never seen this done for libc stuff fot example).
So all you need to do is to find one person who likes to mess with their system (for example, by installing latest versions of the libraries from random PPA's), and you'll end up with poisoned cache for everyone.