To help you decide your priorities some countries now mandate this by law. Eg. all of the EU.

Given the amount of effort startups put into getting accounts, correlating their social media accounts and harvesting user data, and run aggressive analytics, I find it a bit disheartening that the comparatively minimal effort necessary to let them leave is ever considered a waste of time.

I shouldn't be considered a fly trapped in amber just because I happened to want to try a site out, particularly when so many require an email address or facebook account to sign up or sign in (and in some cases, to find out what it is the service does.) Hashing passwords is extra work too. Common courtesy towards users should count towards the minimal amount of work necessary for a site. It's not as if the users who want to leave, now want to stay because you want to hold on to their data.