Heroku Is Currently Down | Hacker News Reader

Heroku Is Currently Down

dashboard.heroku.com

175 pointsiamricks56 comments

Render.com is such a better option these days (and I've been a huge Heroku fan since the beginning). Render has better support and pricing. In addition to great docker support and free static sites, they support distributed Elixir out of the box as well as autoscaling.

Still, they're not immune to these types of outages - I believe they're dependent on Google Could but are moving towards running their own data centers at some point. But so far, uptime has been great for us.

Fly.io also looks promising; keeping an eye on that one.

Thanks so much for the share!

Fly.io's pricing page seems impenetrable to me... I just checked out Render.com and it looks so easy to understand.

I've been wanting to run a Node backend API thing but can't figure out where to put it... been using Cloudflare workers but it kept timing out (going over 10s). Will definitely check out Render's services and $1/mo crons.

I've been a Netlify customer for a number of years now and I'm considering switching over to Render. My biggest gripe with Netlify recently is don't make it straightforward to upload site without using their build system, and there are a bunch of little workarounds to implement to make everything work right.

How is render for static sites? Do they make you build the site with them or is it a simple "here's an NPM package that github actions can consume".

(I work at Render) We do need a Github/Gitlab repo, but we're not wedded to it. So I'm clear, you want the static site hosted on Render but built via GH Actions?

Haven't used render.com for long, but it looks very promising so far, not just pricing wise.

One of the biggest appeals to me is their simplified, docker-compose like approach to infrastructure as code. Even if I only have a few services, dbs, etc. I still like to have it as code under version control, but things like Pulumi or Terraform are way overkill https://render.com/docs/blueprint-spec

There are still various quality of life improvements I'd like to see, but to me they are on a very attractive path

Wow, I was just talking to someone about how I'd love to be able to define all of the parts of our Heroku infrastructure in code. I've been going down the Terraform route but I think it's just too complicated for our team. Render's blueprint spec is pretty much exactly what I was looking for.

Thanks so much for pointing it out.

I'm glad this idea resonates with you. Please take a look and send feedback on docs or features improvements we could make!

I'll convert one or two small applications over to you and I'll be sure to reach out. Thanks!

Hey there, Adam from Render here, I manage our product eng team. Could you say more about the quality of life improvements you'd like to see? Or happy to chat with you or anyone else with suggestions. I'm adam@render.com.

Thanks for the offer! Will write bit more details later on when I have time, so briefly:

Maybe it is less about new/improved features, I strongly prefer a focused product over one that tries to fulfill everyones whims, but more about guidance on for example for a small team what you think is the “best” way for them to integrate render.com into their dev workflows with typical stacks. How to solve the most common needs and what kind of things are out of scope for render to solve and what could be used instead. For an opinionated tool, it could be a bit clearer, what its opinions are. I think lot the information is already there, but it is quite scattered around between docs, feature request pages and forums.

Render had some pretty bad outages (10+ hours) and couple shorter ones just couple months ago, so it's not perfect either. But anyways I have also found it better than Heroku for my projects, it's even easier to setup than Heroku and pricing is slightly better too (especially the free static sites).

(Render founder) We've learned a lot from these outages, most of which stemmed from DDoS attacks. We've since built DDoS protection into the fabric of Render: https://render.com/blog/free-ddos-protection.

Specifically related to this AWS outage, Render can already transparently shift your app between AZs, and we're also working on moving services across underlying cloud providers (GCP/AWS and eventually bare metal).

Ok that's great to hear, thanks for clarifying

Would be great if your service supported JVM applications in the near future. Kotlin on the backend is a thing.

Agreed and noted.

Awesome! Thanks.

(Render founder) Thanks for using and recommending Render. Happy to answer questions. Incidentally, we launched a Heroku to Render migration tool just a couple of weeks ago: https://render.com/docs/migrate-from-heroku

Can you put actual pricing on https://render.com/pricing please?

They are there, but you need to click on each individual component to get the full list.

We're going to update the pricing page so you don't have to click through things as much.

We're looking at Digital Ocean App Platform for this right now. Seems to be a very similar implementation and similar pricing. Has anyone does a comparison they can share?

I created a ticket through Salesforce's "premium" phone support, but the URL of the ticket is hosted at help.heroku.com, which is also down. As of now, Heroku's status page appears green. Our Heroku app is hosted in Amazon East, but appears to be functioning at the moment.

is there a support email? i sent an email via support@heroku.com with title urgent but no reply yet

Sorry no offense, but it's obvious you've never worked in support before.... :-) I'm sure they already know that half the infrastructure is down. Every support request brings exactly nothing, except more chaos.

Auto-responders must still reply "Thank you for your message it's really important to us". At least you'd know the auto-response servers are still up.

People are anxious, if you have a problem in your infrastructure at least set an automatic reply linking the status page or with a short text describing the situation.

Could somebody please explain, how come such a sophisticated PaaS provider as Heroku was not able to properly set up multiple AWS availability zones for their own services? I would think that if somebody had expertise to do it flawlessly, it should have been them.

I don't understand much in cloud devops, I just judge from recurring HN comments saying left and right "your startup should have configured up multiple AZ" when some AWS region is down.

I don’t know know how Heroku use AZs but I do know their platform in AWS EU-West-1 (Dublin) if funtioning fine. Dashboard + metrics all ok, haven’t deployed though. So that’s to say they aren’t dependent on US-East-1 for all internal operations if you aren’t in it.

Edit: maybe the dashboard is just back up and I don’t see it down.

Most heroku users are using relational databases like postgres, and relational databases don't failover transparently. If client apps can't failover to other AZs, why does it matter for heroku's service itself to do so?

Getting geographic failover to work seamlessly generally requires application-level knowledge or at the very least, exotic datastores like DynamoDB or Firebase. For better or worse, that's just not the way most people want to develop.

RDS has this mostly solved. They silently mirror the master into another AZ. If the primary fails the internal routing redirects to the clone.

Tell me, are transactions guaranteed to have replicated before they commit?

If not, you have a major consistency issue during failover. Maybe your workload can accept dropping a window of transactions, maybe not.

They are. RDS multi-AZ replication is synchronous. Aurora is asynchronous, and multi-region replication is asynchronous, but a multi-AZ deployment of the "traditional" database engines in RDS is synchronous. It's rather the opposite of what you said before: if you stick to the old fashioned databases, this is easy to set up, failover is automatic, and no app-level changes are needed.

I think you may have been thinking of multi-region, since you mentioned geographic failover. Multi-AZ isn't really geographic failover.

It is usually some unexpected secondary service that wasn’t known to be in the critical path. Or a hidden dependency on an aws service that didn’t fail over gracefully. I look forward to reading the post mortem.

Trello is down as well (at least the checklists aren't loading anymore). GraphQL requests timing out in the Console. Not sure if it is connected to Heroku going down. Just wanted to add this here in case anyone else is also experiencing the same issue.

presumably because of AWS: https://news.ycombinator.com/item?id=29648286

Weird, my apps are hosted on us-east but they are up, i just can't login to Heroku nor can i deploy to any app right now.

It’s just a single AZ by the sound of it

Yes, but I think it's elastic compute that's down, which hosts a lot of AWS and Heroku's own infrastructure, hence the large blast radius.

And that single AZ took down all of Heroku? This'll be an interesting post-mortem to read.

Your existing apps are likely in the “data plane” path which is more reliable by design of most cloud providers.

Status site: https://status.heroku.com/

I imagine they'll eventually feed us with the tried and tested line, "this outage prevented us from updating the status page".

And we will reply with the obvious response, "run your status page on different infrastructure".

Edit: status page updated

All 3 heroku hosted sites which I look after appear to be up and running normally, despite the heroku console being unavailable. Not sure if auto-scaling is impacted yet.

Edit: all my apps are in the EU region, which is eu-west-1 (Ireland)

What region are your apps hosted in? My apps are still up all on us-east-1

We are also up - hosting in EU. Can't access the dashboard.

All our monitor are green, but we did experience a bit of increased latency for about 30 minutes.

Can't login using their id service https://downforeveryoneorjustme.com/id.heroku.com

Can't log in to their console either. Apps are running. seems the id service is down.

I just noticed that I cannot upload images on Slack. Tried to use imgur to post a link and that doesn't work either.

Noticed this when trying to build an Electron app; atom.io and therefore all related artifacts are down.

I got an error trying to download the git client for Windows... must be this too.

AWS is having issues in us-east-1 again -- are these two services connected?

Yes, Heroku is a (PaaS) built on top of AWS (IaaS).

I am able to deploy now but i still cannot access the dashboard

Got issues on Imgur too.

may be they ran out of dynos

It's affecting Substack.

Reply on news.ycombinator.com