Auth0 Down | Hacker News Reader

Auth0 Down

twitter.com

123 pointsguscost76 comments

Don't forget to request your credit per their SLA[0]. You have 10 days to request your credit, which by my calculations should be 10% of this months' charge. Not a fair trade for leaving us dead in the water for 4 hours, but SLAs in general are worthless.

[0]: https://auth0.com/docs/support/services-level-descriptions

I'm willing to bet we can get at least the 20% return, they're still "officially" down. I bet it'll be a few hours till they say we're good.

Everyone make sure you've got detailed tickets into their service queue. Get your creds!

Their status page is pretty depressing to read since all it really says, repeatedly, is "we're really sorry" and "we're working as hard as we can." You can just feel the abuse the person writing that must be getting from customers. I feel bad for them since they have no power to fix this and didn't cause the issue.

No root cause or resolution yet and it's been 4 hours. Doesn't bode well for getting this resolved soon.

Ha, whenever some 3rd party SaaS tool is down and I am obsessively refreshing their status page I'm always enraged with the lack of updates. But then later I think about what if I were one of the engineers trying to bring things back up. There is nothing worse than being heads down trying to get a mission-critical system back online and being constantly distracted by demands for "status updates."

What is the ETA on getting back online? Well, about 5 minutes longer than it would have been now that you pulled me away from what I was doing....

In my experience,

Good organizations have scheduled updates for critical outages. No one pesters anyone, and everyone knows that a five minute update is expected periodically, e.g. at the top of every hour.

Great organizations instantly spin up a response team which includes a communications liaison, who is technical enough to participate in the reponse, but is not presently leading it. They assist and communicate, either to the PM or to customers directly.

...and other organizations burn through humans until they get a handle on how to be better.

That said, in these days of AWS and GCE, you can go a loong time without running into the kinds of failures that used to cause critical, extended outages.

Yep, there is definitely a way to do it well and having scheduled, regular updates (as opposed to ad-hoc requests) is key. But when "the big one" comes then it is hard to maintain the discipline in that process.

And what is really frustrating is you see an error in a tool you depend on, try something to fix it (such as rebuilding a database index), then you sit back and wait without having any idea how long that process is going to take. Or if it is even going to fix the issue. So when someone asks how much longer, you really don't know. And there is a lot of quick pounding on the keyboard, followed by a long wait period where it feels like you aren't doing anything.

This type of incident is exactly why I dislike identity federation as a service. Yes it's difficult to get right, and you open yourself up to additional risk and technical complexity to do the federation yourself, but simultaneously how many businesses are currently completely down and just sitting on their hands waiting for Auth0's engineers to fix their systems?

But how will you ever succeed if you don't outsource all but your core competencies while your core competencies simultaneously converge on banking/investment/midde-manning?

While this protracted outage has prompted some plans to get rolling with a self-hosted Auth0 substitute, bear in mind that this can't address users wanting to sign in with Google, Microsoft, LinkedIn, etc., where you'll still depend on their authentication service to work correctly, not ban your client ID, and so on.

Yup ^^ this. That and some dang help with managing security audits for our security team -.-. Just can't afford the man power to do it all.

We managed to get a reply from a C level. All we could get out of them was "something to do with our DB, but we don't know the root yet. Our fail-over process didn't work. This will never happen again".

Also, it only took them 2 and a half hours to admit it was their entire system instead of "a small subset of users" lol.

Wasn't their previous major outage because of a bad migration?

I don't think so, I think that it was a combo of malicious intent and some indexes that never got run. I guess you might call it a bad migration since indexes didn't get run, but that seems more like a catalyst than a root. https://cdn.auth0.com/blog/20181128-Incident-RCA.pdf

"something to do with our DB"

Oof. Is there something about what they do that prevents you from having a completely separate second site? Or is this a case where "bad data" is being happily propagated to the redundant site?

As core as the service is, I imagined a panic button that reverted the database for site #2 to some specified point in time.

From what I've seen on some internal email chains, there is a fail-over process for HA multi-region/site, but it didn't work right. Whomp whomp.

This is crazy timing -- my co-host and I just released a Podcast episode yesterday sifting through the details about the Auth0 database-related outage from 2018 ( https://downtimeproject.com/ ).

I'll be curious to see how much overlap or not there is with that previous outage. They wrote up a nice post-mortem back then, so hopefully we'll get another one this time.

Do you have a link to that previous post-mortem?

https://cdn.auth0.com/blog/20181128-Incident-RCA.pdf

heh, smells like its similar. thanks for the link.

Wow, finding out just how many services use Auth0 because of this. Like I can't log into Segment (getting a 500).

I guess Okta did buy something core to the Saas ecosystem

> However, we are still tracking down the root cause of the issue. We understand that this is an issue impacting the entirety of the environment

They still don't know what's going on. This is concerning. Already down for two hours of prime time.

What alternatives to Auth0 are worth looking into? Between this P0 (with no ability to check the status or file a ticket) and the Okta acquisition, I hesitate to continue using Auth0 as the default when spinning up new web apps.

I dunno, Auth0 is still pretty stellar. Its not perfect, but the rate of disruptions isnt very significant IMO.

(famous last words)

AWS Cognito works but it is a far cry from how usable Auth0 is.

Every year or so I check back in to see if Cognito has gotten any better. I hasn't. I'd love to use it, but some very basic things are just not done correctly. Yan Cui has a nice writeup on Cognito [0] that touches on some of its shortcomings.

[0]: https://theburningmonk.com/2021/03/the-case-for-and-against-...

My blood pressure is still coming back down from AWS having an all-day outage during a holiday week last year, because Kinesis had an issue and it turns out their entire infrastructure depends on that.

Agreed! just got done replacing Cognito across the board with Auth0, glad we hadn't pushed it to prod yet.

If you're okay with self-hosting then keycloak or ory.sh seems like viable alternatives.

I'm just getting into prod with keycloak. It's pretty amazing for free.

What's the point of such services? Every web framework worth its salt supports all common authentication schemes and offers an easy interface to write your own.

The target customer is IT departments who want a simpler centralized way to manage access to all of these services for the employees in their org

Auth0 is/was more focused on external customers auth market, and not enterprise use cases.

Okta?

Okta is in the process of acquiring Auth0.

You can try Azure AD B2C.

Keycloak is pretty battle-tested and rock solid although it is a bit of a behemoth operationally.

Outside of that they Ory ecosystem is really nice. We user Hydra, which is not a drop-in Oauth2 server but requires you to write several of the components yourself.

"Keycloak is pretty battle-tested and rock solid although it is a bit of a behemoth operationally."

Just looked. Java, Wildfly, Infinispan, Hibernate + A DB, network multicast if you want a cluster, your own separate load balancer, etc. So, a fairly large time investment if you aren't familiar with all of that.

Yeah, it's not the easiest thing in the world to get up and running but not quite as hard as it might seem at first look. If you are already using k8s then you can use https://github.com/codecentric/helm-charts/tree/master/chart... to deploy Keycloak fairly easily. If you're not using k8s then it is probably more of an undertaking.

Wow. The whole point of paying someone like Auth0 is to _not_ have this happen. This is basically their whole point, is it not? Really looking forward to the post-mortem, but I won't ever forget just how down this thing is right now.

totally

Yup, exactly. We're actively working on ripping them out of our software, this is ridiculous. I knew we should have taken the time to make a fail safe...sigh...

The whole point for me is that I don't want to be responsible for user credentials. I don't trust the security of my app.

If you don't trust your app with credentials how can you trust your app with what those credentials are used to access?

If I get hacked I don't want to leak my users' passwords. The rest I couldn't care less about. It's nothing of value.

Basically every platform and framework has good salting and hashing built in. Usually it's scrypt, bcrypt or agron2.

> The whole point of paying someone like {insert cloud provider} is to _not_ have this happen.

Being in the cloud doesn't mitigate potential issues.

It just means that I can point to someone else when the site goes down and can't do anything to get it back up. Like Auth0 today, or Microsoft a couple of weeks ago.

I suppose the point is that you get one 5h outage every other year _and_ you get to blame somebody else, rather than more regular ones because it's not really your core competency

We all know that all 3rd parties will have outages at some point, nobody's pretending otherwise

Oh lawd its starting to come back up! Pingdom monitor is flapping and we're starting to get some successful, albeit slow, logins!

Yes, looks like it started to work for us too a few minutes ago.

Even their status page was showing a Heroku down page for a bit.

I believe it still does: https://status.auth0.com/

Their own app is down too https://manage.auth0.com/

Anyone down to build Auth0.1?

First RCA is out: https://cdn.auth0.com/blog/Summary_Root_Cause_Analysis_(RCA)...

US-1 downtime: 4:20min. Cause: database query contention

This is the "quick" version. They write "we expect to provide our detailed RCA for this incident by 5:00pm PDT (24:00 UTC) on April 29th, 2021, or sooner."

And their support center is down as well, so the offline ticket form is displaying, but prompts for login and fails on submit.

> We have identified that this issue is coming from our database. However, we are still tracking down the root cause of the issue. We understand that this is an issue impacting the entirety of the environment, and the full Auth0 team is engaged in resolving this.

their dev is up, but prod is down for several hours mid day, INSANE. My users are asking me to have a failover... hard to explain they ARE the redundancy plan! Anyboy else have ideas here on how to explain to users?

Do you mean that your plan was to hope that they have a redundancy plan?

Every SaaS application has probably blown past its SLA thanks to this. Not sure if anyone will trust this site anymore.

They're fast approaching the 95% availability mark for the month! Enterprise customers are already guaranteed 50% return on fees, I suspect they'll be giving out MUCH more than that.

Absolutely insane. Already have a task in to move all our authentication away from them.

Management console went down for us again. 502 service unavailable.

Doesn't seem to be affecting us (UK company)

The outage is affecting the US tenants, the EU ones are working fine.

Still down our whole app is down

Still down for us... 12:49pm (more than 1 hour after starting)

We've been down for over 2 hours and are still down.

just successfully authenticated a few times

The Okta effect?

If anything Okta would help them with this. Never experienced an Okta outage.

This is feeling more and more like a malicious hack.

Waaaaaaaa come on guys and gals!

Reply on news.ycombinator.com