Heroku: We’ve Heard Your Feedback
blog.heroku.com
blog.heroku.com
Well, those, and:
1. Speed. It took days for heroku customers to be told about this.
2. Customers sign up at "heroku.com", the platform is called "Heroku", the CLI is "heroku", everything's heroku, so don't send emails from a parent company (Salesforce), send them from "Heroku".
3. Unambiguous info on what customers need to do. I had to guess based on HN comments whether config vars were accessed. Config vars are 100x more sensitive than code. Comms should be unambiguous and complete, and if incomplete for any reason, explain that (e.g. we don't know yet).
4. I still don't know whether having 1 Github Deploy on my Heroku account allowed unauthorized access to all heroku applications on my heroku account (i.e. those using other deploy methods, like `git push heroku main`). Were all my apps' repositories able to be accessed, or just the one(s) deployed via Github Deploys?
5. I still don't know whether unauthorized access was gained to all other GitHub respositories on my GitHub account, i.e. the repos that aren't heroku apps.
These said, I still really appreciate that security incidents happen and aren't easy to deal with, and there's no obligation for anyone at a profitable company to actually care about semi-captive customers, so thanks to Heroku for the efforts; it's genuinely appreciated.
interestingly, when i log in or reauthenticate, or log in through the heroku cli, I get sent to https://verify.salesforce.com/v1/verify/
They're working on something called "Project Periwinkle" that is intended to remove all Heroku branding and make everything Salesforce branded. Periwinkle being a colour between blue (salesforce) and purple (heroku). No more Heroku signups, you'll need a Salesforce account to use it. No more free tier either.
Heroku has been in the process of being sunset for years now. New features have been banned for years. Only "keep the lights on" projects are allowed. Not that they could do anything with the skeleton crew they have running the platform.
Bob Wise's LinkedIn doesn't even mention Heroku, only Salesforce. Lenora mentions the project here: https://www.lenoraporter.com/portfolio/salesforce
Source: I'm a former employee that left in protest because of this project.
This project Periwinkle sounds awful. Basically thats the end of using Heroku for us. If it remains like this its something to judge from.
It's worth mentioning that this isn't a result of Salesforce acquiring Heroku. That happened 12 years ago when Heroku was next to nothing. Salesforce gets credit for investing in and making Heroku. Why they ultimately have decided to give up on it I have no idea. I hear it's because salespeople had a hard time understanding how to sell it which seems like a strange reason to give up entirely.
That's sad, I guess.
So you had an idea after all?
Everyone thought they were the darling child, at least for the first 18 months.
Yet unfortunately from my experience with large orgs and sales, the sales people get all the control and freely shit on the people actually making the software they sell.
[1] - https://render.com/
[2] - https://fly.io/
[3] - https://porter.run/
I next looked at fly.io, which seem ok for the first two apps, assuming they use the same database. If I ever want more than two apps, it seems I will need to start hosting different apps together, which is the opposite of the headache-free experience I'm looking for.
Porter runs on my own cloud account, so I can't trust it to not cost too much.
Maybe I can get a Kubernetes cluster somewhere (DigitalOcean?) and deploy all my small apps to it, but it sounds like a headache.
I'm staying with heroku.
It is difficult for me to understand why Salesforce is not aware of the strength of the Heroku brand among experienced technology workers, and how much they have destroyed that brand in the last 2-3 years.
Guess I'm packing it up as I'm no longer seen as target audience (its my hobby project platform)
> New features have been banned for years. Only "keep the lights on" projects are allowed.
What???
Eventually they'll get there, but customers should know that's the direction. They'll be users of "Compute Cloud" writing APEX instead of Procfiles someday.
That's what it's looked like from the outside -- that no features were for whatever reason(s) no longer going to happen. But still dismaying to hear it from the inside in those terms.
I think the feature freeze happened in 2018
I don't know what's going on exactly. Those who set up heroku for the first ~5 years somehow did such a good job that they could coast for another 5-10 and still stay on top of what they were on top of.
I have paid apps and free apps (staticman comment processing, and very simple apps I create while following learning tutorials). This project sounds bad! I’ve been slowly exploring alternatives, and was about to abandon my search due to demands on my time. Guess I need to keep exploring.
I did try a heroku competitor recently and my builds failed. And there was no detailed log to show why. So I couldn’t troubleshoot it and I immediately gave up!
As an aside, I was really hoping this post was going to be “we read that one competitor’s blog post last week clearly detailing all the areas where we can improve, and we HEAR you.”
And then drops a link to contact them, via LinkedIn…
LinkedIn is the polar opposite of GitHub. It’s the worst example of social media, from its news feed, to spam invites. And it’s broken every rule in the “be a good netizen” play book, from constant spam, to slurping your email contacts and surveillance to the extreme.
I struggle to imagine a developer saying “I’d like to contact xxx, and I’d love to do it via LinkedIn”
Why not drop your email? Or a GitHub profile with a public email, and readme containing other contact methods, would have been more dev centric.
It does pose the question, what is the most developer friendly contact method??
GET /contacts
POST /message
Interested to hear ideas.
If it means 'easiest method for only developers to contact me' that would probably be some documented REST API.
Either way, I agree the answer isn't Linkedin.
Tbh, I don't a single dev who uses LinkedIn unless they looking for a new role I.e they're certainly not using LinkedIn as a dev-oriented news feed
WaPo and AWS are both owned by the same person. That doesn't mean both (have to) charter in the same territory.
I agree with the general sentiment of your comment, but LinkedIn is the worst? Really?
I can think of a couple social media platforms that add less value to society.
LinkedIn has helped democratise the recruitment landscape significantly across many parts of Africa; we rely on it extensively.
Not wanting to pick a fight here, but that’s one helluva hyperbole I just couldn’t let slide ^_^
By the way, how many senior devs and cofounders are left at Heroku Bob? Why doesn't it show up at dreamforce anymore?
Heroku never seemed to want to integrate, but instead be the "cool kids", those who just do not have to worry about the enterprisy stuff such as automated backups, DR, high availability, enabling Java, etc. Everything they did was great, everything "Salesforce" sucked, yet they never made any money before the acquisition. The acquisition was a waste of money in first place (IMHO).
The founders did not even try and so they left.
You’re correct that the lack of real integration with SFDC was a large part of the problem though. Though as far as SFDC acquisitions go Heroku was cheap. More expensive ones had far worse outcomes even with integration.
What we knew how to do at Heroku was build a great platform for developers to launch apps. We never claimed to know how to make that model work for enterprise. In fact we were awful at trying.
Seems like Salesforce got exactly what they bought. Not sure why they’d be surprised about it.
Want simple? Use a build.sh
Want a bit complexity? Use a Dockerfile.
Their docs are really copy paste for 99.9% applications. Whoever in that company made the decision to invest in docs was right on the money. It made switching to them a much easier choice. https://render.com
Also the reason Django has always been so approachable (particularly in the early days, as it is orders of magnitude more complex now) and therefore popular
Older still, IMHO also why mIRC scripting was so much fun despite its shortcomings. The help file had literally everything you could want to learn about the language and it almost begged to be read. It's how I got into coding
Here's where I'm starting: https://render.com/docs/migrate-from-heroku
To any render people reading this, it appears to have been an issue with the migration plugin and the Heroku-18 stack. Manually deploying the app worked fine.
Creating postgres cluster redacted-db in organization redacted Launching...⣻ Error failed creating the Postgres cluster redacted-db: Timeout on CreatePostgresCluster.app
Every attempt fails. There's no way I can run this for $250/mo on AWS. Now I have to pretend I'm a sysadmin and use Hetzner.
Is there a way to assign an existing Env Group when manually creating a new service? So far I've needed to create and then go back in and assign.
The intention presumably is to be able to say they communicated this while (intentionally) limiting the amount of awareness that's spread about this.
Wow talk about terrible timing.
Fail it's because I am new; Succeed then it's my superior capability; plus, the learning rate during crisis is at least 10 times higher than the peaceful time, let alone the nerves and mental fortitude where peaceful time would never train one for.
I launched several startups on Heroku over the last decade +, and feel they have gravity on trust. Some of the best devs I have worked with.
Now the attackers had access to encrypted environment variables?
> We also wanted to address a question regarding impact to environment variables. While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets.
If the threat actor had access to any of the systems that use the key, they may not have needed to. Even this statement isn't clear that they couldn't have done it, but suggests that they don't think it's true...
This is really bad incident response messaging.
When they started sending out password reset emails, they should have explained why.
Not only when people started complaining, and the media picked up on the lack of transparency.
Turns out the master database with encrypted username/passwords got leaked and encrypted environment variables were also leaked but it was like pulling teeth to get them to answer whether or not these happened or even admit that it might have been possible. Presumably more than this was also leaked but so far they haven't said anything on that. Env vars were the biggest concern on everyone's mind.
They gave the absolute least amount of information over the longest period they could muster.
The problem wasn't the hack really, it was the lack of transparency in the response.
See: https://twitter.com/jacobian/status/1522782890957819906
WTF!!!!
That alone is disastrous enough, they should be reprimanded for this. Are there I'm sure, class action lawsuits happening?
How much of an impact will this have on Salesforce? I mean imagine the data from that alone would be immensely valuable.
If it comes out that the hacker did get to unencrypted env vars I think it's game over for Heroku. Nobody should trust them with sensitive data.
It means they don't know. They must say "we are confident" otherwise.
I still don't forget how they handled the Intelligent Routing fiasco from back in the day.
Can someone more familiar with an event like this tell me what they are working so hard on? I imagine securing the vulnerable service and resetting various credentials doesn't take that much work.
Also, generally, it’s a “fog of war” scenario, where you can have so many unknowns to work through in a compressed time period, and sometimes there’s an active attacker and they get a vote, too.
For starters, you have to document everything. At the very minimum the legal team for the company should insist on this, if no other measures, just as a CYA move.
If an attacker had broad access, it's entirely possible an all hands on deck approach is required to help identify (and document) what systems were compromised. Yes, you definitely want a team working on patching the hole ASAP. You also need a team hunting for any possible persistence. Another team probably involving standing up brand new "safe" systems and failing over client systems over to those running the patched software. While that happens yet more people may start doing audits of what was compromised on the original systems.
I've seen incidents where 50-80 people were pulled in to work on an incident at a company of about 150 employees. Depending on how well-funded your SoC is they can cut that number down substantially.
Our team realised heroku was going to take weeks and we’ve replicated our “review app” development workflow with GitHub actions that clone apps on PR, push code and rebuild them on push and destroy them on PR close. It’s not as seamless as the heroku GitHub integration but it’s good enough for now.
Unfortunately, git lfs isn't supported for CLI pushes, haven't found a way that works yet.
We’ve been lucky that this entire event has, theoretically, not touched us as we never connected GitHub. That may change as more information comes to light.
We’re still strongly considering moving to AWS, and are in the process of getting quotes from vendors.
Glad I didn’t even realize there was github integration!
That way org admins can see the requested permissions and control exactly which repos are permitted. GitHub OAuth apps are an absolute nightmare to audit or control.
They should go work for render.com
They were the gold standard for rails deployments back in the day when rails was popular—which was important since rails was actually pretty hard to host on a server compared to say PHP.