Heroku was Down
Incident: https://status.heroku.com/incidents/2402
Update: our apps appear back up after 23 minutes total downtime. Others are reporting applications still down.
Update: it appears most or all services have been restored.
Update: our apps appear back up after 23 minutes total downtime. Others are reporting applications still down.
Update: it appears most or all services have been restored.
I've seen a few get to the front page.
But when we're right we're right.
Of course, they won't. If they host is on someone else's then that might look bad (tacitly saying that a competitor is reliable and might be up when they are down) and if they hive off an extra copy of some of their infrastructure there will still be single points of failure either accidentally, by human error (someone somehow messing up both segments at once), or by design (possibly through management trying to save pennies when they noticed this extra bit of infrastructure on the balance sheet).
I worked there and I vaguely remember something like this but it's been a long time.
If I had to guess this is probably a DNS issue (it's always DNS).
As an aside, I spoke with our AE in early January on where they are going in dealing with AWS unreliability. One would think they would have a good answer. They don't.
I recall at least one of them I could not deploy new versions of my app though, which is pretty bad. I don't believe I had any downtime at all due to heroku platform outages in 2021. i believe you if you say you did though!
This particular outage definitely seems to be of a rare level of severity.
Didn't bring my deployed app down though! No user-facing outage for me.
This time my app was down for about 35 minutes.
It's much better than it was 5+ years ago. Back then they had almost weekly downtime.
I mean, yesterday was a good day. But today's the best you've got if you didn't do it then.
Heroku and AWS are organizations no?
> While there are not currently any specific credible threats to the U.S. homeland, we are mindful of the potential for the Russian government to consider escalating its destabilizing actions in ways that may impact others outside of Ukraine.
like what
I'm just posting what would lead somebody to that conclusion. Nobody is definitively saying anything right now.
Might be a good time to run: heroku pg:backups:download
› Warning: Our terms of service have changed:
https://dashboard.heroku.com/terms-of-service Apps: Yellow Data: No known issues at this time. Tools: No known issues at this time.
=== Availability of Common Runtime apps 2022-02-24T16:50:47.249Z https://status.heroku.com/incidents/2402 investigating 2022-02-24T16:50:47.249Z (1 minute ago) Engineers are looking into reports of connectivity issues to Common Runtime apps in the US and EU regions.
It's always DNS.
We'll have to wait and see I guess.
<td class="bb pad04 top center" style="width: 32px">
<img src="/images/status0.gif">
</td>
Cache freshness checks involve a lot of headers, which take up way more bandwidth than one unicode character.
/s
But, this really happened in 2017.
Metrist monitors via bots
[0] https://downdetector.com/status/windows-azure/ [1] https://downdetector.com/status/google-cloud/
I did figure, okay, a 30-second-to-load status page probably means my app outage is a heroku platform problem.
(Also an indicator the status page is sharing too much platform with the platform it's supposed to be reporting on? Also in this case an indication that the platform problems are pretty deep?)
Interestingly, my app logs (via papertrail, which is still up) show that some traffic is getting through continually through this current outage, although I can't (and my monitoring app can't either, which pinged me).
Link here to latest snapshot:
No issues noted there.
As I typed that out I remembered that they handle DNS, load balancing, and databases, so I guess any one of them.
While Heroku is busy fixing the issue if you scale your app to a single dyno (assuming it can handle the traffic) it should restore availability.
However, while I could not connect to my app, nobody I know could connect to my app, and my monitoring service could not connect to my app for a ping... my app logs showed that some traffic continued to connect throughout the outage. So it was not entirely universal. And was clearly a routing problem of some kind.
(It took a few minutes to get this link to work for me)
We can access our logs and the application is running, just no incoming requests.
My app was down about 35 minutes total, according to my monitoring.