If you are attempting to index the whole internet then you are doomed to eventually download something illegal.