Duo Outage | Hacker News Reader

Duo Outage

status.duo.com

We use Duo at work for 2fa. I am laughing my ass off, I literally can't do anything -- can't get on the VPN, can't get into my emails, can't access company services. They locked everything down so hard, they've literally chained themselves to a radiator and tossed the key out of reach. We didn't receive internal communications about the outage yet -- and my bet is it's because whoever's in charge of that, is locked out due to the outage.

I would never do duo 2fa only. It sucks if one of your factors does not work. Most of the time I always try to use two 2fa apps.

TOTP tokens don’t meet alot of compliance requirements. You usually need a PIN or crypto device with a dedicated solution.

Like the popular password vault says in the name… 1Password for everything. If everything can stand alone in 1Password, it ain’t MFA.

This is also an unfortunate coincidence, as today is Georgia Tech's first day of classes, and the institution's Single Sign-On (SSO) for almost all its services, such as the Canvas LMS, registration, and the Bursar, is based on Duo's service. Right now it seems to be having some issues and is timing out / not logging in.

However, I'm not sure if GT's Duo service is self-hosted or is hooked into Duo's service "in the cloud".

I am reading this, slacking off, because I couldn't get into Georgia Tech's Canvas. Judging by the status.gatech.edu message, it seems like changing the instance of Duo that responses is a quick fix.

I wonder if it’s more causation than coincidence.

Single Sign-On, single point of failure… maybe having lots of different logins wasn’t such a bad idea after all.

There are a lot more other problems that happen when each service manages their own authentication. The move to SSO has been in response to problems that existed then.

Same for my kid picking up info for the beginning of their first year at Univ of Maryland.

Millions of students coming online, textbooks are digital now, SSO all the things. Perfect storm.

University of Kentucky too, first day of classes, no one can get into anything they aren't already logged in to with a valid cookie.

I spent 20 minutes trying to figure out what new cookie I needed to grey-list for the half dozen redirections in the M365 auth flow to not bork before I thought to check if it was generally broken.

Great success.

I'm actually taking a course there right now, as part of my pointless masochistic love of continuing education in stuff I like knowing to be well-rounded but no one will ever give a crap about professionally, and I logged in fine. Duo's push notifications seemed to stop working, but it lets you fall back on the TOTP passcode, which continued working.

Duo seems to have had lots of success, selling into higher education. To add to the other schools mentioned in this thread, Oregon State, and Stanford both use it too. And lots of schools are starting class today, which might be what contributed to their downtime.

To be honest, their useR experience is pretty slick compared to most two factor solutions. I am happy for them, (and their users) that they have been successful, selling into large organizations – good user experience usually isn’t found in large enterprise software due in part to the principal agent problem.

Ohio state is another ~60k duo users this week.

University of Idaho checking in, first day of classes as well.

And, yes, Duo is normally a great solution for us.

University of Illinois Urbana-Champaign starts today too, another 60,000 or so users.

> selling into higher education

Just want to add another data point: Dartmouth was a relatively early adopter, and while a bit annoying, it chugged along just fine.

My duo workflow was figuring out I can request an SMS OTP and read the OTP code as a notification on my laptop instead of unlocking my phone and confirming via the app. Hopefully they get volume pricing on texts.

This kind of workflow is probably against your workplace's rules. This bypasses one of the protections meant through something like a Duo app locked behind a device password screen; which is that even if your laptop is logged in and your password manager is running, a bad actor still couldn't get into protected things if they don't know your phone passcode.

If someone is relying on Mac+iPhone users disabling a convenient and enabled mostly by default feature then they’re doing it wrong.

university of waterloo as well, although thankfully classes start in two weeks for us

UC (Cincinnati, not California) first day of classes is today too, I always hate that out IT doesn't allow 2FA except Due. I always wanted to request an exception, but my previous experience with them refusing to allow IMAP access to my email because it is not secure!.

Not often I see UC mentioned on HN. What program are you in?

Hi, yes it is rarely mentioned.

I am in the physics program

Did you notice that 2FA went away around 10 am? I worked on the team that made that happen. Its back now that the outage is over.

We do have an app that lets faculty exempt students from 2FA, but it's mostly for students who need the exception when taking tests etc.

You are going to see a big change in the tech at UC very soon. The old guard is getting the boot.

> Did you notice that 2FA went away around 10 am? I worked on the team that made that happen. Its back now that the outage is over.

Yes, altough I did not get affected that much. I was just trying to renew overdue book from the library system. But I'm sure ~45k student got affected harshly in their first day of the semester.

> We do have an app that lets faculty exempt students from 2FA, but it's mostly for students who need the exception when taking tests etc.

It would be better if there is an option to allow graduate students who do research (PhD candidates) to be treated differently as they are not students anyway. And Duo is annoying. I understand that this is not something that will happen specially with the reputition of UC IT department (sorry but you problably know)

> You are going to see a big change in the tech at UC very soon. The old guard is getting the boot.

This is something I have been hearing since I joined but without the old guard getting the boot part. Each year with increasing student enrollment, we can't even provide stable internet connection. I still remember two years ago the outage of the auth server for wifi system on the first day of classes (after covid) and this stayed the case for almost a week.

I miss Tech. Still using Oscar and T-square?

Ugh. Those memories.

I was among the last to have to use dial-up from off-campus (which required a specialty ISP @ 26.6k or some awful speed). Fortunately, they upgraded their system while I was there, so I could use the “much” faster DSL. Still not as nice as on-campus internet.

This doesn’t mean GT is offline. There are a couple dozen Duo server #’s and this only affects DUO1. If GT is on DUO5 or DUO19 or DUO25 they’ll be fine today. It does raise the question of how to provide continuity of services if your server is offline though.

I thought this was one of Google's messaging apps.

It was a video call service: https://en.wikipedia.org/wiki/Google_Duo

Wow, never heard of this one - how many messaging/chat applications has Google created now?

~17, depending on how you count: https://www.theverge.com/2021/6/21/22538240/google-chat-allo...

Don't forget Google Photos...which also has chat for no discernable reason.

Is it really discontinued yet? If anything it's an ongoing naming blunder with no end in sight.

There are two products called Google Meet now. The web interface for the former Google Duo is duo.google.com, just rebranded to Google Meet. The former Google Meet still exists on meet.google.com. Both also have their own Android apps - one called Meet and other Meet (original). Both products have a different set of features and neither completely replaces the other.

duo.google.com redirects to meet.google.com on my iPhone.

Google Duo was rebranded into Google Meet.

The original Duo app was named "Meet (original)" on Android phones, but the old Duo icon remained until you launched the app (so it could inform you about the name change). Then the Google Meet app was introduced. This resulted in phones having three icons ("Duo", "Meet", "Meet (original)") for the same service.

Same here.

Same name, different app.

I thought this was Duolingo, a language learning app.

And before that, I thought Duolingo was a crowdsourced translation app, that doubled as a language learning app.

Same. For anyone wondering:

Duo is a multi-factor authentication/single sign-on platform owned by Cisco

I was always curious why Duo cloud 2FA was chosen over traditional offline 2FA codes in education.

User experience is nicer: click confirm prompt from lockscreen vs. having to find and open app, select correct site, and enter code.

From the student side I end up carrying an extra 2FA app that only works with my college vs. Google Auth (and similar) that carry my other 15 sites.

> click confirm prompt from lockscreen

This is also a huge vulnerability that has been exploited.

https://www.theregister.com/2022/11/03/mfa_fatigue_enterpris...

Offline 2fa codes seems like a nightmare to roll out to an academic campus

Indeed. They are generally understaffed and salaries are very low so they're very lucky to get any "1x-5x" developers who stick around long enough to understand the infrastructure. Outsourcing as much as possible makes a lot of sense in that environment, it does create major single points of failure but "roll-your-own" would likely fail more often anyways.

It's simply so much easier.

Okta offers a similar feature. So much easier to click a confirmation on my phone than to scroll through dozens of 2FA codes (some of which might be orphaned).

well apps like 1password makes it easy to fill those 2fa codes from computer and phones easily.

Absolutely. For me, I don't think I would use anything other than 1PW/Yubikey for 2FA codes.

But not everyone uses such technologies, and a certain percentage of population is going to find the hurdle to adopt these technologies/apps too high.

So, not for us, but I understand why they do it.

This implementation sounds better. Though for me I still have to manually input a code from the Duo app (that doesn't auto refresh after code entry since it's not time based).

Having the do the manual entry and the lack of refresh is a choice of your security team/administrator. Duo supports push notifications and auto-refreshing TOTPs.

Source: I used to work for Duo.

People really don't understand 2FA codes. Imagine trying to tell thousands of students to get the code from their 2FA app (Which app?). What happens when a student goes home over the summer and gets a new phone, but doesn't transfer the app info? Duo offers a level of management that other apps don't. If a student is struggling, you can send them a text with a direct link to the app they need to download. You can temporarily bypass 2FA from the Duo console. For the longest time, it was the only 2FA app that offered any kind of management. Okta has it now, too, but most higher ed already has a different SSO provider, so switching to Okta just to get 2FA management (And I'm not sure it's as good as Duo's) is probably an impossible task to get off the ground.

It’s two-step auth, not MFA.

And the proud Illinois tradition of some mission-critical service crashing on the first day of class continues.

In this case, it is an external service. However, I also suspect that the Duo outage is probably shielding other on-campus services from load surges that would probably be causing them to get crashy.

I guess I don't know how we could ever prevent such incidents. Given that the first day of classes is a well-kept secret /s.

I love that the status page is being updated regularly.

But I have no idea what the difference is between DUO1, DUO2, etc. through DUO73. I feel like they should have a better way to clarify which users are affected.

They're all identical deployments of their whole stack. They shard customers onto the deployments to reduce blast radius. In your Duo admin dashboard, look for "Deployment ID" in the left pane.

They published the postmortem, and the penetration into colleges is hinted at as being causal.

> Increased load on DUO1 due to significantly increased adoption and simultaneous peak usage across multiple larger customers led to authentication failures.

https://status.duo.com/incidents/rw7g0q7ztj8f

FWIW I just logged into my university portal and Duo worked just fine. Perhaps it's not widespread?

no issue here in australia

I bet both of duos users will be bummed to hear about this outage.

It's mandatory 2FA at the college I went to (where, coincidentally, classes are starting today)

College-induced software!

It's almost like all the college and university CIOs are choosing the same software platforms. Now why might that be?

I work in post-sec and this is very common practice. There are few key players that tend to capture the majority of schools in the States/Canada for specific tech solutions. Blackboard/Canvas/D2L for LSMs, Shibboleth for SSO, Duo for 2FA, Cisco AnyConnect for VPNs.

tech solutions in the field tend to be incredibly low risk given the size and make-up of the anticipated users (enterprise services with thousands of employees and tens of thousands of students). For public institutions, there's the added element of public sector risk avoidance.

To be clear, Shibboleth is often self-hosted and usually the grey-beards understand how to maintain it. It's been around a long time and is very stable/robust and at least as unlikely to fail as Duo/Cisco (which are overall fairly robust with rare enough breaking failures). OTOH, rolling their own 2FA would likely create points of failure that rear their ugly head more often, not less often.

Shibboleth is kind of an outlier here, due to its age/maturity and position as a very old-school piece of foundational tech that got implemented when academic IT salaries were quite a bit easier to live on than they are today.

The disparity between tech salaries in academic institutions and FAANG/SaaS corps has grown immensely in the past 20 years. Most of the people who do the real work at academic institutions have been employed there for 25-40 years. Most of the young people can't stick around for long because they need to earn more money to build a stable life.

I wasn't criticizing any of the companies I listed.

Makes sense, but I still loathe the clunky stuff I end up using!

You make it sound like it’s some big conspiracy but the boring answer is that nobody gets fired for choosing an industry leader.

Duo has positioned itself as an industry leader in MFA and is one of the safe bets when implementing that feature.

This isn’t exclusive to schools, either.

This is Cisco's Duo used for MFA and SSO and not Google's discontinued video app.

Yep, and my company uses it as part of access to all of our internal applications including our AWS console.

Not a super productive morning

I didn't realize this Duo was affiliated with Cisco. Now it makes sense it's so popular

It's impacting me, but I was a little surprised to find out that Duo was "popular" enough for this outage to make it the front page of HN.

I posted it and honestly I am too

Never underestimate the number of college students hiding in the online shadows

I've used it while working at two major companies everyone's heard of and used. Definitely seems to be popular.

Reply on news.ycombinator.com