It really depends on your individual tolerances. Before I started on my trend towards paleo I was heavy into dairy and I've determined that it contributed to a stomach cramping problem.
Coffee is a bean, as is cocoa. I can manage pretty well with beans most of the time, but they're doing something that I can't pinpoint yet. It's an iterative thing, and when I cheat it's an opportunity to learn more about how those things affect me.