ProtonMail Is Down | Hacker News Reader

ProtonMail Is Down

protonstatus.com

156 pointsthesurlydev71 comments

I mean, no one cares if accessing mail is down, but it looks like their MX is down, the status page reports incoming mail has a ‘partial’ outage.

I don’t care if I can’t access my email for a while.

I very much care if email sent to me isn’t received. They should just be dumping all mail received at their MX’s to maildirs or something if it’s this bad and process them properly later. Losing data is not ok.

SMTP is literally built with this scenario in mind. As long as their servers aren't incorrectly reporting a success the sender will (or at least should) retry.

Fortunately, retries are built into the spec (rfc5321). Typically, you retry for 3 days.

> They should just be dumping all mail received at their MX’s to maildirs

Not that it makes all that much difference, but this would break the encryption model for non-pgp inbound emails which are meant to get encrypted with the users public key on arrival.

I have no doubt that manual interception of these messages would be trivial with a court order anyway.

"Proton Mail mobile apps are are now operational again, and so is Proton Calendar."

https://nitter.net/ProtonSupport

also RSS feed:

https://nitter.net/ProtonSupport/rss

I'm kind of curious what the source of the outage is.

It looks like most of their services are down, including their incident status page¹. The number of things I can think of that so many different services & are not a massively catastrophic failure is pretty small. Though that may be a lack of imagination on my part.

1: https://status.protontech.ch/incidents/195

I'm seeing their status page up now.

But they are down hard otherwise.

maybe the status page is therefore also down, misleadingly showing its last pre-down state

They have another status page that currently reports everything is up (it isn't).

https://status.proton.me/

And another that says that some systems (many systems) are experiencing issues/in a state of "partial outage":

https://protonstatus.com/

"An error occurred on client"

Maybe it's the same hard drive problem a lot of others (including HN) had.

Doesn't have to be catastrophic failure. Most people just assume that such services are architected properly to be HA, but they usually aren't. They also aren't operated as HA usually, so things like a config change are rolled out globally to everything at once, and then everything breaks and they can't roll back, and you have total outage. Very common.

...and of course, they might not be down at all, it might just be (example) a route from the west coast got fucked in a BGP table so the west coast can't access it, and if that's where the majority of their users are that's all the comments you'll see.

What is HA?

high availability

High Availability

A bit nitpicking but even with a proper implemented HA solution there would be a service interruption.

> The number of things I can think of that so many different services & are not a massively catastrophic failure is pretty small.

Like, for instance, a simple DNS or BGP misconfiguration that takes the entire network down?

Cloudflare (June 2022):

https://news.ycombinator.com/item?id=31823132

OVH (October 2021):

https://news.ycombinator.com/item?id=28849319

Facebook (October 2021):

https://news.ycombinator.com/item?id=28752131

Google (June 2019):

https://news.ycombinator.com/item?id=20077421

We don't know for certain, but you may have missed Rogers [0] - which brought down cell phones (every function but wifi), debit transactions, and 911 services.

[0] https://en.m.wikipedia.org/wiki/2022_Rogers_Communications_o...

No ones it too big to not fuck up BGP or DNS from time to time.

  https://protonstatus.com/incidents/195#update-201
  45 minutes ago : Our engineering team is working on a database performance issue which is impacting Proton services

Relational database woes. :( Probably re-indexing or slow queries or something. This is why companies with an RDBMS as a SPOF need a good DBA.

Maybe they installed a whole lot of SSDs on the same day 4.5 years ago.

2 things to ask.

1. why don't people selfhost email more? mailinabox is pretty painless to set up and would give you infinitely more security/privacy whatever you want. 2. why is email deliverability still an issue? why do you need to use third party email tools to ensure your email doesn;t go to spam? i am not concerned with newsletters/ads/ that sort of thing but a replacement to gmail/protonmail??

i have personally gone with miab like last year and beyond the first few weeks of gmail putting my emails into spam(then having to inform the other user to unspam it), things have been painless. i get to occasionally spin up the ssh to update the server but its fine. backblaze b2 backups ensure i am somewhat safe(er). can it be better? yes. is it ready to be used in production, aka a replacement to your regular paid/free email? you be the judge but i am sold on this idea

You replied question 2 with question 1: we don't self host email, -because- of deliverability issues.

why are they in the first place?

Spammers

Deliverability is an issue in both directions:

- when you send emails from MyWebsite.com, you want to make sure that Hotmail, Yahoo, Gmail, Proton Mail, etc users get your email.

- when you receive emails to your hi@mywebsite.com email, you don't want spam

V1 of email providers ostensibly solved this problem using lots of data as well as "supervised" machine learning (e.g. you telling the provider that a specific email is spam).

Self-hosting seems awesome if you're only handing out the email to a few people (probably a few hundred max?), you don't think it will be widely shared, and it's a slightly unique email that makes it hard for a script to guess (e.g. "hello@mywebsite.com" will be easily guessed but "john2@mywebsite.com" may not be)

- because google would prefer you use gmail so they can train their data models on you

- google would also prefer to have a monopoly on all email so that g-suite is necessary to reach your customers.

- because your IP address was at one point an e-mail spam server, but, that only really matters because of the above two reasons. Any nit, however irrelevant, will be used used to beat you into using gmail.

All anti-spam systems have non-zero false positive rate. It is hard to fix all FP in advance, but relatively easy to do this on case-by-case basis. When an FP affects some large sender it gets fixed. When it affects small senders - it is ignored because Big Corps don't care about small fish. They would be happy if all small mail servers will disappear and the market will be controlled by a few big companies.

You can self-host email all you want if you intend only to receive. If you want to be received, by O365/Gmail - that's another story.

It is technically possible to self-host email and send mail to O365/Gmail. They don't block 100% of small mail servers, only a fraction of them. But if you out of lack and their AI decided to block your IP, contacting their mail team is form very hard to impossible.

> why is email deliverability still an issue?

Mostly what I've found as the main issue here is that popular blacklists will blacklist entire IP blocks instead of individual IPs, so you have to not only have a clean IP but also be on at least a /24 with all clean IPs. This can be hard to manage when relying on outside hosting. And unfortunately, trying to get around it by using a cheap/free relay like Amazon SES or SendGrid doesn't work, as they fail to keep their IPs clean (if they're even trying).

It has gone down again today.

Their statement yesterday:

Yesterday we unfortunately experienced technical difficulties which meant that Proton services were offline intermittently throughout the late afternoon and evening European time. During this period, incoming email, outgoing email, and push notifications were also temporarily delayed. As of 2:45AM Geneva time, all emails and notifications are caught up, but we are continuing to monitor the situation in case of further faults.

We can confirm that no data and no emails were lost. Due to our redundant infrastructure, emails that were not immediately delivered were queued and automatically redelivered. We are still investigating the issue, but what we can share so far is that the technical difficulties were caused by a software update that was conducted over the weekend which adversely impacted the performance of certain systems. Unfortunately, this issue only appeared under under high loads which we were not able to fully simulate in testing. Furthermore, because this was a major software and operating system upgrade, there was no way to roll back the software update.

This result of this was an extended period of intermittent outages and performance issues while we tried to work around the issue. We have now put a number of safeguards in place, and will continue to improve the reliability of our infrastructure to guard against this type of issue in the future. We apologize deeply for this issue and thank you again for your understanding. Our engineering team is currently conducting a root cause analysis and for transparency, we will be sharing results on the Proton blog after our investigation is complete.

I thought it was my phone or vpn, I restarted it many times. Where would I be without HN :-)

3 days ago, not very far!

https://news.ycombinator.com/item?id=32023848 [Hacker News was down]

Geez if both were down at the same time I might have no option but to touch grass.

And if grass isn't available, I guess there's always myself.

funny that you say that: "HN is up again" https://news.ycombinator.com/item?id=32026571 2 days ago

> Where would I be without HN :-)

Reading the regular news like an idiot!

>there was no way to roll back the software update

oof

>During this period, incoming email, outgoing email, and push notifications were also temporarily delayed

They say no data was lost but I'm not sure how they can know a sending mail server didn't give up trying to deliver a message to Proton

It would be surprising if a the SMTP retry timeout of the average sending server was less than the duration of this outage.

This is not to say there is no cause for concern, but SMTP was built for scenarios like this, at a time when these things happened far more often.

This has always been my concern as well.

I wonder if there are any global standards, requirements recommendations about m the minimum retry count or timeout.

Looks like it's down again: "Some systems are experiencing issues" [1]

------

[1] https://protonstatus.com/

Has to be related to this maintenance announced a couple days ago[1]

> "We're conducting a short database intervention on Sunday, July 10th, 2022, between 9:00 - 10:00 am GMT+2. During this time, users will be unable to access all Proton services for 2-4 minutes. We apologize for the inconvenience."

[1]: https://twitter.com/ProtonSupport/status/1545698028895584257

I've been having trouble with iCloud all day too. (Try Apple ID > Password and Security on your iOS device). It surely can't be related.

A minute ago StackOverflow was also throwing "Oops! Something Bad Happened!" to me on many different pages. I wonder if that's somehow related or just a coincidence.

Rather ironically, the site they link to for the incident status updates ( https://status.protontech.ch/incidents/195 ) is not loading for me: "The connection has timed out"

Sorry about that.

I just moved the domain for my small business over to ProtonMail (where I already hosted my personal email). The fact that I did that guaranteed that it would go down. :)

I was logged-in -- and composing a message -- a few minutes ago. First there was that cannot reach backup server message followed by a bunch (don't you just love technical jargon?) of messages saying "Could not..." and then lost the message when the site itself apparently went down. (Well, Firefox was seriously unhappy...) I was ready to blame my ISP again but no, this one is on proton mail. (It's pingable and traceroute is happy.)

I'm really glad that we didn't move our business over...

Seems like it’s not yet completely sorted… despite being impacted, i feel for the guys steering through this outage and bending backward! Good luck guys!

Was down 5 minutes ago, but is now working again for me

I'm on it now, must be back up?

Agreed. I'm back up too

Still down for me :(

Correction, its down.

Slower login, but works

Down hard

its like blackhawk down but w/ proton mail.

I mean the connection drops immediately, unlike the other times when it spins forever

good thing I just bought a 24 month pro sub last week...

It sure was an excellent purchase -- one of the most secure services available world-wide, operated in the EU with strong privacy and completely outside US jurisdiction.

You weren't able to collect your emails for a brief 5 minutes, but you'll live.

Switzerland is not in the EU.

Hello from Switzerland :) We are not in the EU not just "technically" but as a fact. None of these agreement covers the exchange of private emails or metadata with the EU.

But as a fact with have to comply with EU-Laws -> Schengen.

Irony is the people most likey affected by a protonmail outage would be political staffers and strategic PR firms who would suddenly need to use messengers subject to disclosure.

As someone taken out by the Rogers outage last week, I'd be concerned the infrastructure of western powers may be too fragile to project power in a seriously contested conflict.

Just remember, clicking on refresh and searching the net what is going on want make the problem go anyway faster. Few of us urgently depend on email just this hour. An even if you do setting your alarm to some suitable interval and just doing something else is much more productive. Just act like the Arab cliche taxi driver who takes the hands of the steering wheel when it gets tight and say Allah will steer (No insult to anyone, I have never visited any Arab country. Probably the story is not true anyway, but a nice anecdote to accept things you cannot change.)

When hackernews was down recently I wanted to report it as big news in our company chat. But it was rather late here and I had already closed all work stuff for the weekend. So I first slept and when I woke up everything was working again. Oh yeah, I missed to share the scoop with my colleagues, but I guess I'll manage without any damage to my professional reputation.

Edit: Reading post-mortem afterwards is a different story. If it's well written instead of some corporate communications emptyspeak one can learn something. No mistake happens only once.

>Few of us urgently depend on email just this hour.

I agree that you shouldn't be manically refreshing your email every 5 minutes, but my mail service being down for a few hours would definitely be a major inconvenience if I needed to get a 2fa code during that time. Same if I needed to look at my email to view my tickets for an upcoming event.

That sounds like an fragile setup if you depend on a single email account to get 2FA codes. I have them configured typically on my office workstation (reachable remotely if everything works), on my laptop, on my phone, and I have the static backup codes in my password manager which is also distributed over several indpendent machines.

Of course if you have to respond to incoming tickets within a guaranteed short timeframe that gets a problem if nothing can come in.

Make your risk assessment in advance: Can I live without it for a while, in the worst case even until I have an alternative provider? If so then just do so when an outage happens. If that's not acceptable for your case you need to have an independent backup before the first outage.

It's a misfeature using e-mail for 2FA login codes. TOTP is far more reliable. Any service doing e-mail only 2FA should be called out and questioned. Never assume a user has access to their email at all times.

Right, I must admit I had never heard of email being used for 2FA codes. Is that some kind of standardized protocol or something completely homegrown?

In the far past (and still today with 2 banks) I use hardware/smartcard based solutions. But everything vaguely IT world seems to use TOTP. They typically advertise it as Google Autenticator, but I use different FOSS implementations on both PC and phone and they have worked 100% smoothly.

> Right, I must admit I had never heard of email being used for 2FA codes. Is that some kind of standardized protocol or something completely homegrown?

If you don't have 2fa explicitly set up and browse with VPN enabled, you'll frequently get sites that want you to authenticate via email. Amazon does this, for instance.

Reply on news.ycombinator.com