If you're a small company and making a good faith effort to acquire express consent from your users for data collection, follow best practices to secure that data, and allow users to delete their data (even if it's a manual customer service driven process) then I don't see why you have any reason to be afraid.

You're not opening yourself up to litigation from random 3rd parties -- they can only file complaints to regulators who will decide how to respond.