The Decline of Heroku
infoworld.com
infoworld.com
I know a company that used to spend enormous amounts of money on heroku. Like, seven figures a year. The company was growing quickly, billion dollar valuation, raised hundreds of millions, etc etc.
One day the company passed the billing threshold that put them in the highest heroku tier, and someone in the org got an email from Heroku. The email was congratulating about all the incredible new customer service they would get now that they were spending more than $X a month (dedicated support 24/7, priority access to new features, that sort of thing).
The email made its way to some higher up who...hadn't realized just how much was being spent on heroku.
They switched to AWS a few months later.
Most expensive support email I've ever seen!
Somehow this spaghetti-behemoth actually worked, and I spent the next year fixing bugs and adding features. We even modularized it enough to support deploys to either Mesos/Marathon or early Kubernetes. It still sticks in my head as an example of where somebody did everything "wrong," yet they still did it, and it ended up being extremely valuable to the company.
I think nowadays with k8s being what it is, and with all the tooling around it and increasing industry expertise—not to mention managed offerings—Heroku is going to have a hard time.
there's just not that many folk trying to tame deploys on k8s via gitops. flux2 is the rage, it's all over the alpha geek's efforts[1], but it's usually used by someone carefully authoring a fairly complex Helm file, then building out a significant Flux2 HelmRelease object (ex: [2]).
there's a bunch of other tools[3], & i'm frankly not familiar enough. but this idea of having a bunch of source that can deploy itself, simply, is still extremely rare even among the alpha-geek #gitops types. i'm sure some of these tools better match the simplicity of the Heroku model, corresponding branches to environments, which makes so so much sense, but so far i feel like such attempts are still basically unknown.
heroku's really simmered it down to something that made extremely natural sense. huge props to that. too too much of this effort had to go into creating buildpacks & supporting language environments very very carefully very actively, that ability to stealth-containerize an app & not even notice is so much of the special sauce that makes this a hard, hard & eternal problem (because langauges/envs keep changing). there's still a lot of ease of use to Heroku that's potentially will be underrated and/or lost by the oncoming generations. i have high respect for how operateable Heroku is.
[1] https://github.com/k8s-at-home/awesome-home-kubernetes
[2] https://github.com/onedr0p/home-cluster/blob/main/cluster/ap...
Heroku is for companies that are willing to pay for convenience.
However, once you have developed a workflow, going back in developer experience is hard, and hacks like these become common. (I saw a team do a similar tool to avoid paying for Cloud Foundry.)
We used Heroku for years and were spending around $2k per month (having increased from $1K a few months before) when we migrated the last stuff to GKE and our spending dropped to nothing instantly. I was pretty surprised to have nobody at all reach out to us to see why our spending had changed... Now we spend 7 figures on GCP instead.
Heroku solved a lot of problems incredibly well but they seemed to lose any ambition to be more than a salesforce deployment platform a few years after the acquisition and as k8s and other solutions have gotten better they just don't have a super compelling proposition now. I still use them to spin up side projects though as they still provide the best completely out-of-the-box experience.
Heroku is a great product, there is no denying that. However, relying on startup/small teams is kind risky. Some startups fail after awhile, those who succeed sometimes "grow out of" Heroku to keep their cost down.
I think what Heroku needed is a plan or two that can keep the both end happy, instead of offering a plan that eventually driving successful teams away.
Engineers always complain about speed and price.
If you start comparing ec2 micros to Heroku dynos you already don't understand what Heroku offers.
Generally I will see small to medium startups that hire 3 devops engineers who barely have the bandwidth to setup a fully fledged deployment system e.g. pipelines, red/green light deployments, backups, stability etc
This all works from the get go with Heroku.
Most developers scale resources too early and don't go for easy wins like slow queries and any other latency related problems.
How many times have you seen developers recommend upgrading from Ruby to Go because of speed? When all they had to do was add some partial indexes or something to their database.
I've shown non technical CEO's in small startups to scale, rollbacks, view logs and restart their servers themselves.
I will always pull aside CEO's and tell them that their engineers want the best for the company, but that is only from their vantage point, the likely don't even know the companies inner financials, so how can they tell you something is "cheaper".
Focus on your companies core competencies first, get users, make money then let your engineers build their dream devops setup.
(If your startup does some mega processing jobs like ML then yeah you will want to use something like EC2)
I recommend most startups use Heroku. It takes a _huge_ scale to make moving to ec2 make sense in most cases.
Everything runs on the instance; a nightly job backs up the database to s3.
Deploying new code takes 15-20 seconds; if I want logs I can ssh in and look at them. The old ways of doing things work better than ever in 2021, thanks to cheap vertical scaling.
That said over time as the hosted k8s solutions have gotten better Heroku has comparatively become less compelling, plus while Heroku can get to a large scale, you will eventually start to run into limitations with what you can run inside a dyno which would force you to move to something else. Since you'll have a 12-factor app though, it's relatively easy to migrate, the hard part is finding a new platform which doesn't feel like a massive downgrade vs Heroku.
Nah, and that's a bit of a hassle. Yesterdays data will do in a pinch (they'll have to manually rekey a days worth of data and some might get forgotten, but that's what stocktake is for).
That said, it's been running continuously for 3.5 years without downtime so far.
> What about if you get an influx of traffic and suddenly need to scale beyond what a single server can do (which is admittedly pretty high).
This particular app serves a small business, and I'm quietly confident they'll contact me before adding more than 100x their current staff overnight. It's currently running on a t3.micro, but I'd expect upgrading to a p4d.24xlarge with 1tb ram, 8gb NVME storage and 96 vCPUs will handle it if they do.
> How about logging
If I want to read the log files, I ssh in and read them. Sometimes I even scp them to localhost.
> deploys of a live copy of your site for each feature branch
I don't need that for a project with one developer and one stakeholder - I can just share my screen and walk him through what I'm changing.
If you didn't know all that, how long would it take vs `git push` to deploy your website?
Startups live and die by the ability to rapidly test ideas and go with the ones that stick. Heroku removes the absolute need to have devops knowledge.
Heroku is great, if you just want to start with something and show a proof of concept. As soon as you are doing more than just that you will probably get the first steep bills (at least from my point of view). We actually migrated from Heroku just a bit more than a year ago.
Most companies I know wouldn't really profit from k8s much if at all. A handful of VPS's setup half automatically using Ansible and a few clicks will be probably just enough. You will know, when you need to scale out to many servers when k8s is probably the better approach. You will probably have people doing just that full time at that point.
I have a side project on AWS that I deploy with capistrano, runs on a rails deploy that I configured by hand, and requires approximately 5 minutes of "dev ops" maintenance a month from me. Most of that is upgrading packages.
Switch that over to docker and terraform and who knows what else, and I wouldn't even have time to think about the project, but for the parasitic costs of the deployment infrastructure.
I'm 100% certain that n00b devs would mock me for my old man ways, but the old ways can scale quite far before you need anything else.
Our project has almost no traffic but the data processing part requires plenty of RAM. Literally with a few concurrent users it just won't work. And if autoscaling was enabled it'd be too pricey.
So I think it also depends on the use case. There are better PaaS alternatives.
PTSD flashbacks to when we had to rewrite a Javascript monolith in Go microservices because we didn't know about Dataloader.
I wanted to index billions of very small documents. Billions was in the "phone us" pricing category.
Switched to the smallest cheapest VPS I could find and that did the job.
I am not saying, hack everything together. If you have no clue about infrastructure, please use Heroku. :-) I am just saying, you can get very far with rather great performance, stability and security by using a VPS and some backup. You will learn some useful skills along the way and frankly, you don't have to care much about performance because even the second or third cheapest tier will be way more than what Heroku can offer at multiple that price. I mean "Performance L" in the Advanced tier of Heroku is ~ $500/ month with 14 GB RAM, I cannot see how much CPU/ disk etc. At OrgPad, we would probably need that one. The whole infrastructure, with disaster recovery and everything, costs so much less, we could run our infrastructure for a year with that much money.
And yes, languages are mostly fine with today's HW. Look at SQL/ IO first.
There is also something I like to call the Ruby on Rails dilemma. Rails is a beast when it comes to rapid prototyping and small teams. I don't think anything else comes close. But as your team and system grow, all the conventions that let you speed ahead in the beginning start holding you back. Now you are digging through Rails code and doco trying to find ways to override defaults without destroying everything else.
Heroku is the PaaS version of Rails. Small projects are super easy to get going. You don't get a better starting experience. But as you grow, all the conventions of Heroku that helped you launch with ease start becoming month-long expeditions for work-arounds. Worse, Heroku isn't open source. You can't look at the code to understand your issue. It doesn't take long for someone to lose their minds and spend a weekend creating a presentation on the benefits of moving to another platform.
[Edit: so what I'm saying is the characterization of heroku is only good for very tiny teams/projects/etc has not been true for me.]
YMMV of course - depends largely on what you're building.
Now, I'd say Render is the modern version of what Heroku was. It's super easy to use for Rails/Laravel/Node/etc, it doesn't break clustering of Elixir apps and static sites are free.
I get their website as the first result for both queries, both in my signed in profile and in a clean browser session over a vpn.
Interesting! I've been experimenting with Gigalixir for a Heroku-ish experience that supports Elixir clustering... I'll have to give Render a look and compare the two.
Compared to other cloud providers their web app is a marvel of user experience. Monitoring, depolyments, scaling, docs, configuration, adding new services...It's all a breeze on Heroku. I dont't want to write scripts to orchestrate containers, I want to easily deploy, scale and monitor my app.
I spent 5x more time in AWS EC2 and Beanstalk and it's still daunting.
For me, the only downside of Heroku is its high price.
Yes, it was a huge up-front investment of time, but it will serve me forever.
New startup? I can have the infrastructure that will last us a couple of years up and running in like an hour (that's including setting up CI/CD and code review tooling and monitoring). Most of that time spent making the cloud accounts.
I sell my services and this is a huge enabler in my work. I don't necessarily see the benefit of outside contributors. As an educational tool it also kind of doesn't make total sense without context of how things work in the various cloud services.
So while I kind of want to, it's also sort of my competitive edge and interaction with the community is a drain on my time without adding much to my bottom line. That is to say, I've opened pieces of this previously and all it did was bring tons of demands on my time without any paying clients (and I already get enough inbound that I don't need more name recognition).
As a side note though, Terraform has probably made more of a difference to my career than any other piece of software. Shoutouts to Hashicorp, really. Consul, Vault and Packer also have been majorly useful but to a lesser degree. They've really put a lot of money in my pocket and that's the best feature any software can have. :D
Most of my work comes through people/companies I've previously worked with and recommendations. As much as possible I try to set things up so they rely on their cloud provider for support, but most questions are either simple enough that I share my advice freely or can charge a day rate to do some work. I insist on working with at least one person to own the accounts & code/state (which can be like paid training for them). They may be paying for my knowledge/experience, but I prefer to leave them artifacts to reference after I'm done.
I charge reasonable contractor rates per diem.
I also have a full time job, so while I'd be open to taking a retainer to be available for immediate work, the fee that I would charge for such a contract is absurd. Basically more than my full-time employment.
The script itself is probably 20-30% of the time, 70% is learning the underlying system/technology. Once you've got the learning out of the way, the scripts start making things super easy/fast.
Heroku is still the best for smaller projects (and medium sized depending on the type of the workload) and devs that are not fully devops.
Sure, if you have a previous project where you can copy pasta your Terraform and Kubernetes setup from, you can start your infra just as fast, but otherwise heroku app:create and git push heroku master is still way faster to start hustling.
Everyone compares invoices between e.g. AWS and Heroku, but do they ever take into account the cost of the ops team when they switch to AWS? If the bill was 7 figures on Heroku with no devops, how many senior devops does it take to get the same spend with AWS? How about productivity loss if deploying is not as easy as it was before?
I think saying something is dead because it doesn't work in all use cases is not fair.
A point-in-time restore really saved me once (allowed recovery of important data after an accidental delete), but I am interested to know if other think I am over-blowing their significance in a production system? Are there other strategies which make PIT backups less important?
Is your 2021 roadmap public, or is Canny your official roadmap? Really interested to see what's coming soon :)
[1]: https://feedback.render.com/features/p/deploy-docker-images-...
Btw, I've been working at Canny for the past year and was pleasantly surprised to see your feedback site!
Last time I tried Render I couldn’t find a way of easily deploying an existing image.
[1]: https://www.digitalocean.com/blog/introducing-bring-your-own...
Private registries likely in 2021; we're actively hiring! https://render.com/jobs
Neither dokku, heroku or any other PAAS that I know if supports this and its really important for some of the more interesting features the beam offers such as pupsub without having to use redis. In my startup, we're using horde to manage a cluster of dynamically spawned children. We setup a kubernetes cluster on aws for this. Today I would look at using render instead.
I fully expected to be on a more 'modern' platform, but virtually every other provider requires at a minimum a Docker config. Docker has its purposes, but I can literally push an arbitrary Go/Python/Ruby/Node/etc. repo to Heroku, and it will be hosted on the Internet successfully. It's still SO much faster. As a comparison, I did need to get one project running with Docker and it legitimately took 16+ hours of developer time to get source control authentication working as a part of the build process.
None of that delay is a huge issue when you have limitless engineering resources, but for someone starting from scratch Heroku remains (bizarrely) the fastest tool out there.
Probably best you stick with Heroku
Same here. After dealing with complexity of building an app the last thing you want is more complexity. And this time of a kind you're not comfortable with.
Obviously a lot of people were ready to pay to Heroku to hide that kind of complexity from them.
And Heroku was never a great solution once you reached a reasonable level of scale because it became expensive and your Heroku infrastructure became subject to unexpected issues in resource sharing within shared instances. Shared instances are a problem across all cloud providers but pure-AWS-sans-Heroku is easier and has more options to switch to private instances.
developers should never be comfortable with operational complexity. It's like saying javascript is a good language because you got used to the quirks and warts.
The pricing model isn't ideal but it has a place for some projects.
However, for personal use, if you have a few small (web)apps, it's way too expensive compared to a DigitalOcean or similar VPS.
For example, 3 applications with Postgres DBs cost 3x(7+7) = 42$/month, whereas it's entirely possibly to run this on a 5$ DO droplet.
In fact I do exactly this using Dokku, and have several small apps running in the same $5 droplet.
I think that is the problem. On the outside it has not changed much and thereby loosing traction and customers.
They have added things but apart from pipelines quite a few years ago nothing revolutionary has been added.
The only thing I can think of is that they made it harder to use when they changed the pricing for their free tier to be per user and not per app.
When you tip over that threshold from "default dead", that is, non-revenue focused activity won't kill your company in 6 months, then and only then do I think it makes sense to switch over from Heroku to something else.
I still use Heroku all the time, and recommend it to clients who are starting out as well. (Just like how I recommend buying a $10 theme for design instead of wasting money fiddling with a designer/design team).
You need to determine product market fit, pricing, and distribution first.
For personal use it was excellent way to spin up an app and see if it is useful etc. That part I had to stop doing because they changed the free-tier pricing model. So I had to sunset most apps, keep a couple on the free-tier and Hobby-tier, but with no room to add new apps.
For work use it is useful for demos and some small apps. But the main problem has always been scaling Heroku apps beyond a single monolith. Most work projects split and scale across a lot of apps/services etc and that stack part was never easy with Heroku.
So for my personal apps I now have an overkill of a K8s cluster (or 2, maybe 3...) that I chuck all my new apps in. As someone with k8s skills that is ok, but obviously I know that Heroku did 1000% other things in the background that I now have to do myself, and pay for...
We want deploys that are reliable, transparent, that can roll back in face of issues, that support gradual blue/green strategies, etc. That's what makes an overall 'platform' good or bad.
Surprisingly, that specific problem isn't very often isolatedly tackled: too often is coupled to a whole platform, a pervasive set of assumptions (you use containers/dynos/VPSs/etc).
AWS CodeDeploy seems to fit the bill well. But AWS being AWS (with a history of leaving certain products neglected) I just don't trust the implementation itself. I cannot see its changelog, issue tracker, assess its internal quality, etc.
So I would really love to see some open-source CodeDeploy clones, with a scope as small as possible and the highest possible internal quality. It could be a game changer.
Recently I've moved to K8S on Digital Ocean and I could not be happier.
You can do others (eg West Coast US, Japan, Australia -- it's all just AWS regions under the hood) but to do that you have to be in a 'private space' which is effectively a dedicated clone of the Heroku management infrastructure.
If you were paying a few hundred dollars per month you're now going to be paying a few thousand. IMO Heroku is shooting itself in the foot with this price structure.
Although even whitenoise is still ok if you're going to set it up with a CDN as the frontend for your static assets. If you want simplicity then you do it using one of the CDN addons.
Whitenoise is not a limitation of the heroku platform, it's really a symptom of suboptimal documentation.
Heroku is perfect as it is. Let's not mess it up.
https://www.digitalocean.com/products/app-platform/
Are there other PaaS providers with Buildpacks support?
After talking to >200 founders, basically every startup wants the Heroku developer experience but AWS is just giving way too many credits for anyone else to compete. Some companies literally aren't paying a dime on cloud hosting until year 2.
The "under construction" page gave me a mild case of nostalgia for 90s Geocities. :)
Edit: nevermind, found the installation shell script right in front of my nose on frontpage!
Edit 2: I think you have issues hosting your installation script:
curl https://powertools.dev/install.sh
curl -L https://powertools.dev/install.sh
Can't someone support the Procfiles (if I recall correctly), undercut prices and still have plenty of room to be excellent?
I don't know of another PAAS that directly supports Procfiles, Buildpacks etc but I don't think they are the parts is what prevents people from migrating anyway.