Hacker News Reader

Comment by ericpauley | Hacker News Reader

This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake.

In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at night, but clearly that's gone now.

View on HN

Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

Having worked at Heroku and had a large part in building Heroku Postgres I do not recall this explicit policy, and it seems very squirrelly to me. Maybe this came in as a policy in recent years and it is the case, but still seems like hiding behind a policy as opposed to doing right by customers.

You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and you would still technically be hosting the database.

Even taking a dump and emailing it to me feels like a safer option here.

There were better answers here for sure. If the honest answer is we just didn't feel the effort was worth it for this class of users at least own that.

Yeah ideally hold onto a backup for say a year, if the owner hasn't come and downloaded it after a year can then assume that they don't want it.

It's not that easy. Those databases might contain personal information which is protected under different privacy laws like GDPR, HIPAA and others and Heroku/Salesforce can not simply store that for longer than agreed upon and Heroku cancelling the account enables the retention period as the customer agreed upon as part of the T&C.

The account was not cancelled.

I wonder if you guys delete email addresses from people who ask to unsubscribe , or when I tell t you to delete my information, according to gdpr

Probably not, because it’s ” difficult “

Many avoid it, but once they have to deal with a (ex-)customer's lawyer they learn what is even more difficult.

Having been through a SOC2 audit: this wouldn't fly. It's on the checklist of issues that you get hit with regardless of what kind of company you are: when customer accounts are terminated, the data retention clock starts ticking.

You can pick an arbitrary time frame for retention, but whatever you pick, you have to communicate to users, and you can't just change it on a whim. Normal customers want this clock short. They don't want you to retain their stuff after they cancel.

The customer account wasn't terminated, the free DB being used by paid Dynos was deleted without any input from or notification to the customer.

I highly doubt normal customers want this clock short when the cancellation is not customer initiated.

Data retention policies are written by compliance people, not product people, so distinctions like overt, deliberate cancellation and "cancellation for nonpayment" or "abandonment" or "discontinuation" usually aren't captured in them. I'm not saying it's great that Heroku deleted these databases; I'm saying: the description given upthread, that the databases were deleted because of contractual requirements, is super plausible.

Plausible, sure, but that doesn't mean it isn't a cop out. There are many things that Heroku could have done to prevent this, like delaying the disconnection when they don't have confirmed delivery of the notifications and/or when they are connected to paid dynos/accounts.

Thus didn't happen because of the contract, but because the people implementing this transition didn't give a crap.

I'm just telling you that what the commenter upthread said rang true. I'm not offering an assessment of compliance regimes.

If you start disconnecting the database, in the same way a user could (on most services?) to keep the database around and idle, there's no way that counts as any kind of deletion. So do that if you need to and you really want to give customers 90 days to react.

Compliance people are product people too (even if in disguise), and obviously (?) vice versa.

Silos lead to issues like this. Cross-functional teams cost more, etc.

Not so much, no.

If compliance affects any user-facing property of the product, then they are, by definition.

Onboarding users/partners would be easier without making them read/click all those consent checkboxes, etc.

That's like saying everybody in a product company is a product person. No, it really doesn't work that way, at least not in general.

The simplest test is "who are you defending"?

Ideally, product people defend the end users. Realistically, compliance people defend the company from getting screwed in an audit and ultimately sued by the government.

They’re accountants in my experience.

It feels like it would've been better to have written things such that if it's attached to a still active account that's paying for other services things would be permitted to work a bit differently, but of course the relevant clauses will have been written long before anybody anticipated a new owner deciding to turn a bunch of things off, so under the circumstances at the time of writing it's unsurprising and not really the authors' fault that they didn't consider that.

I had a vague thought for a moment about maybe being able to give customers the option to opt in to an updated set of terms that would've been better designed for the situation ... but since the original Tell HN author didn't get any emails about this at all, presumably they wouldn't've got that one either so even if (and given 'vague thought' I'm not claiming any particular level of 'good idea' to it) that had been done it presumably wouldn't've helped them anyway.

It seems to me the root problem here isn't so much the compliance initiated policies as a complete failure to take 'make sure people have been adequately warned' sufficiently seriously given the potential consequences.

Deeply unfortunate.

(read that last sentence in understated deadpan dry en_UK to get an appropriate read on just how impressed I'm not by the communications cockup)

Of course it’s plausible. Literally anything to do with contracts or legal in a corp like Salesforce is plausible.

But this isn't an account termination. They deleted the data of an active account, without informing the user.

Why could they not turn it into a read-only database without access from the Heroku apps instead? Then it'd just be a routine change to the service offered, would it not?

I don't think there's anything in SOC2 that says you can't have multiple tiers of deactivating an account. I think the crucial point here is that Salesforce is requesting the termination, not the customer, so any of their obligations to be responsive to customer data deletion requests are moot (unless the customer comes back and actually requests termination).

Create a passphrase, derive the key, encrypt the data dump with the key, send the passphrase to the customer, delete the key. Presto, you can keep the data as long as you want without any privacy implications, since only the customer could decrypt it. Of course, this assumes working communication channel between the provider and the customer.

> Even taking a dump and emailing it to me feels like a safer option here.

I genuinely had to read this twice to get the intended meaning.

as someone who doesn't use heroku, so disregard my opinion:

i'd probably prefer feces-by-email to surprise database deletion

There is a legal difference between a company policy and a contract with a customer.

Please. You'd just need to ask if the customer is OK with 90 days instead of 30. Done.

The company has no commercial interest in doing that, though.

It doesn't sound like this would have helped in this particular case since they were unable to contact the customer.

I was replying to a bullshit claim that they cannot retain for more than 30 days no matter what.

That's hiding behind the T&Cs instead of owning their decision not to even try because there is nothing in it for them.

So you're in favor of companies breaking their terms and conditions at will? I think that would cause quite a lot more outrage and problems.

You did not read my comments, did you?

The T&Cs are an agreement between the parties. That agreement can be changed at any time if both parties agree. So they just need to ask.

But if they make an exception for one person, they're opening the doors to complaints and possibly even legal action from others who want the same change. So making an exception for one person isn't really making an exception for just one person. Instead, it's a large process that needs careful legal consideration.

So you're in favor of companies breaking their terms and conditions at will? I think that would cause quite a lot more outrage and problems.

Do you really never get e-mails from various companies every few weeks telling you that the terms of service have been updated?

I think I get one from eBay alone every other day.

Those can not be compared that easily. There are many changes you can do easily. Changing data retention periods impacts guarantees customers may have given in compliance to privacy regulation like GDPR, HIPAA etc. and thus is a substantial change to the customers in areas with such regulation.

Contracts can be amended when all parties agree on a change.

An exception for a single client might fly for a small company, but it's not really a _simple_ exception when you're a large company (like Saleforce). It needs careful legal consideration and possibly means changing the general T&C for everyone.

Are you allowed to inform paying customers that you are going to do this? This is my primary complaint here. I don't understand how this oversight happened. This is going to cause an enormous amount of time and energy to recover from this.

> Are you allowed to inform paying customers that you are going to do this?

I can't be the only one who's basically completely blind to emails from major companies, including SaaS providers, because they're so fucking spammy that the SNR is like 1:99. Notifying me by email, for one of these places, is functionally the same as not notifying me at all.

[EDIT] Sorry, didn't mean to imply the parent wasn't paying attention, just that I'd fully expect a very high percentage of their users to miss the warning in all the noise even if they emailed everyone—even if they emailed them a couple times, actually. That's the cost of every company sending out tons of "join our online seminar on [product]!" and "hey, look, it's our newsletter you never read!" and "it's time for our weekly TOS modification!" emails.

I'm not blind, they didn't send one. And they admitted to me that they did not send one.

Well, that's even worse, of course.

That's a major failure on their part. I did get several nagging emails from them.

Even if they did send it, email isn't guaranteed. What if it went to spam or was otherwise mishandled or missed?

> ...because they're so fucking spammy that the SNR is like 1:99.

This 1000x. I signed up for an SMS gateway service last year. Just for my own hobby use, nothing major. I gave them $10 to start service, and they charge like 2¢ or something per outgoing message.

They have like 180 different prices for 180 countries, territories, provinces, parishes, cantons, prefectures, etc. Every week one of those prices changes, and I get an email notifying me of that. I tried to turn those off in my preferences, but they refused. I can opt out of marketing, weekly digests, and "tips and tricks"; but I can't opt out of pricing notices.

So I added a rule on my end to hide those. I totally understand where they're coming from. They can't NOT give pricing change warnings. But at the same time, in the flood of constant notices, there may be something major I will miss.

It would be nice if they instead gave me the option to never spend more than $0.XX per message, and return an API error if an attempted send fails for price threshold violation. Then the spam wouldn't be needed.

Shit, that'd even let them optimize their pricing. "If we double this rate, 25% of our customers will start erroring, but factoring in message volume of those customers and assuming they all leave us, it'll still be a 60% increase in revenue from this product."

I use heroku at work at they definitely sent out a ton of emails. It also said they were going to delete them right above all third party plugins (even if you weren’t using their database service)

Right, but in this case - as confirmed by Heroku when contacted - something went wrong at their end and the emails weren't sent in the first place.

The fact that the failure to send a single warning email to this particular customer wasn't both detected and considered a five alarm fire level bug under the circumstances is the thing that moved the decision to sunset the free tier from sad to disastrous for the user who posted the Tell HN in the first place.

Using the fact that a customer was shown exit as the basis for destroying their assets don’t look great to me, at least on surface…

That's peak idiocy and the product of lawyers taking over the assylum.

If you enable daily backup those are nuked too?

That would be interesting to learn. Backups will be just an additional surcharge rather than a "real backup strategy".

If I'm understanding you correctly, the 30 day policy is one that Heroku chose to put in the contract. Engineering might have fought the terms, and yes they need to be followed once set, but it seems totally fair to blame Heroku for creating the limitation in the first place.

When they were written, short sighted acquirers yeeting the free tier was likely not something the people writing the relevant clauses were even considering as a possibility, and honestly it's such a ridiculous decision from a commercial perspective that I find it hard to assign blame for not foreseeing it.

Plus, it would all likely have worked out fine if they'd emailed the customer a warning or three like they intended to do - it was the failure to do so combined with the failure to detect and remediate the initial failure that sent things down such a dark path here.

This sounds like the legal equivalent of looking the wrong way through a telescope.

Whoever fostered that naive interpretation was a nitwit. If they’re an actual lawyer, they promoted an intentional, mutually harmful unilateral reinterpretation of an agreement and should be sacked.

Cowering behind T&Cs like this is intellectual bankruptcy. There’s always another solution. The law is not a programming language.

> We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

Contracts with some customers, surely? You could have the default be 90+ days, then those customers whose contracts specify a shorter timeframe get that shorter timeframe configured on their account instead. You could give the customer the choice at signup, and let them change it later using the settings console. If their contract doesn't specify a period, send them a notification that you will be changing it to 90+ days, but telling them they have the right to object if they disagree with that.

Every single executive in charge of this decision of how to handle precious customer data at Heroku feel be completely ashamed and take a long, hard look in the mirror.

The phrase "aren't allowed" supposes some regulatory agency forbidding an action. When it's your own internal policy that contradicts the action, the proper term is "won't".

No choice? That's just the way it is, it can't be helped? Did God himself come down and decree that Herokai *MUST* only hold data for 30 days? Did the FBI come in an threadten to charge your executives with sedition?

Yea, no. You decided to make the decision for contracts to be that way. The fact that you "fought hard" but that decided on the 30 day retention anyways means that clearly the opinions of engineers don't matter and that the company is completely captured by the lawyers and out of touch executives. It hardly inspires confidence.

It also doesn't at all address the fact that you failed to contact an apparently paying customer that their data was about to be nuked, contract or no.

I assumed the value prop of being on Heroku in 2023 was purely "not having to do the pain in the butt to migrate your legacy app that you made on Heroku in the pre-2010 era" - in other words, I would be really shocked if Heroku got significant new business or any growth at all, now that it's just expensive AWS with some basic CI integration points.

I also assume Salesforce only bought them as a cash generator and has no interest in investing in it. So if they saved bottom line from this move, that's a win for them.

(Feel free to correct my assumptions if I'm very wrong)

"Pre-2010"? Heroku only launched in April 2009. So I would assume the vast majority of current customers are not actually from a "pre-2010 era". Heroku didn't even have a postgres add-on until November 2010 -- the salesforce purchase went through in Dec 2010. In fact, the heroku "golden years" were mostly during the early salesforce period. (Heroku put matz, creator of ruby, on staff in July 2011; there was no Heroku API until 2014!). (source for timeline: https://www.heroku.com/about)

Heroku is still quite a bit of value-added over "AWS with some basic CI integration", people chose it and still choose it because it requires a lot lot lot less in-house expertise and management hours than AWS for most kinds of standard apps. (I'm not totally sure what AWS services/architecutres you are thinking of when you say that; but I'll say: for pretty much all of them.)

(Heroku def has more competition in that space than it did 10 years ago, opinions differ on the relative merits)

AWS of course already existed when Heroku was launched, so if you do consider them just "AWS with some CI integration" then I guess they would have been from the start? What would have made that true now if it wasn't then?

I think Salesforce thought it would somehow be more "synergistic" with their other offerings than it has turned out to be. That they'd get Salesforce customers on heroku when they needed something beyond the "no-code" tools Salesforce already provided, or that they'd do better at converting heroku customers to salesforce customers than they have been. It does seem to be true that salesforce stopped really investing in new heroku features or improvements some years ago, and seem to be looking to minimize costs while continuing to collect revenue, I agree. (Sometimes I'm not even sure how much they care about continuing to collect revenue...)

Sorry, my memory on the dates was wrong.

> it requires a lot lot lot less in-house expertise and management hours than AWS

Yeah, you're not wrong there. I do worry though that it's still too complex to really trust someone with no devops bg to be fully responsible for, and at the same time, it's so outmoded (just meaning, it's not that popular now besides for legacy projects) that you aren't going to find a lot of good candidates who are expert with Heroku compared to expert with AWS.

So to me, Heroku is a gamble that you can get by indefinitely with only what you(r existing eng staff) know how to do in Heroku. Which for some projects, maybe that's a decent gamble for the payoff of easier admin in the short term.

> if you do consider them just "AWS with some CI integration" then I guess they would have been from the start? What would have made that true now if it wasn't then?

Yeah good point -- Let's see if I can try to explain my claim better. I think in 2009 using AWS was completely inelegant in every way, it was pretty barebones and basically just "Rent an EC2 server and do whatever you want, good luck." Heroku had a unique (at that time) approach which made cloud hosting much simpler and more abstract. AWS today at least offers a few ways you can host an application on AWS while doing less server maintenance than you'd have needed in '09. Infrastructure As Code is also something I think has a lot of popularity today that wasn't really a thing in 2009, and Heroku if I remember correctly, isn't really about that. So there isn't as much repeatability on that platform, it's more "click around and build your stuff out in the GUI." Which again, is cool in the short term to bootstrap, but can be painful at a bigger scale.

I used Heroku’s free tier for small projects/prototype. I continue to use their cheapest offering for new projects because it saves me time. If a project grows, I can reevaluate my choice. I don’t think anything in Heroku is a pain to migrate away from. I would miss the intuitive/familiar UI.

Check out render.com, it's a solid alternative.

We currently use render. Its excellent, but is missing a couple key features: like Heroku's continuous rollback protection. If they offered this they'd be head and shoulders better.

(Render CEO) We're actively working on Point-in-Time Recovery and hope to launch it in early access soon.

Think we (Crunchy Data) has been mentioned a few times below and have near feature parity with Heroku Postgres (only thing missing is dataclips coming this quarter). We've got folks that migrate the DB to us then use a whole host of other options including Fly, Railway, Render, etc.

> Salesforce on bought them as a cash generator

As an early Heroku employee, this gave me quite the chuckle.

Same - wasn’t an employee but used it often in ~2013-2015 timeframe. I’m pretty sure the valuation was many many many multiples of any sort of revenue based metric, wasn’t it?

Yep.

Also re the timeline, the acquisition was announced in 2010. So your own usage was already a few years post.

Sorry, I should have guessed that wasn't accurate. But like, surely after they stopped doing any investment in the platform, didn't it start to generate profit easily? How could reselling AWS resources with a premium UI not be profitable?

I just imagined Benioff going "Cool, we can buy this and continue to collect the checks every month, while not developing the UI or integrating it in any way into our business." Basically the same thing they did with Exact Target.

Salesforce is like the Midas touch of destruction.

render.com

Reply on news.ycombinator.com