The "safe harbor" includes more requirements than responding to take-down requests. In particular, the service provider must "not have actual knowledge that the material or an activity using the material on the system or network is infringing", and "in the absence of such actual knowledge, is not aware of facts or circumstances from which infringing activity is apparent".

In other words, though you don't have an active duty to police your site, if you do become aware, and don't take action, you're not protected. Similarly, if you set up something with the sole purpose of distributing copyrighted material, and try to hide behind a "gosh, we never knew that kind of thing was going on until you told us", you're probably not going to get a court to believe you.