IANAL, but depending on the nature of your service and specifically how the content is collected, you may qualify for DMCA safe harbor protections. This means that if you remove things in a timely manner upon request nobody can sue you. This is how Google caches the whole internet without getting sued.
That's all legal mumbo-jumbo. The real world answer is that some people will get mad regardless of the law, so take their content down and apologize. Follow robots.txt guidelines. Don't post takedown replys a la PirateBay. Generally act sane. If you do all of the above you'll probably be ok.