US Senate’s email system melts down in face of security test and reply-all chaos | Hacker News Reader

US Senate’s email system melts down in face of security test and reply-all chaos

politico.com

> Senate staff were instructed to respond to the email, which outlined a potential threat for the drill, with their status and location on Capitol Hill or off campus. But the thousands of simultaneous reply-alls that ensued first clogged up inboxes before staff were unable to get into their inboxes altogether.

That's why we test! Imagine if that happened during an actual crisis. Give them a break - that's the point of testing, to discover these glitches.

Note that another negative outcome is that high value targets (e.g., Schumer and McConnell) are emailing every Senate employee with their location, not to mention the email chain also provides aggregated information on where crowds of targets can be found.

Also, email ain't so secure. A sophisticated enemy - and I'm pretty sure the US government actually has some of those - would love that information. Heck, you can find locations just by locating mail traffic.

> The Senate’s email system, run through Microsoft Outlook

It's 2023 and someone is still writing that?

I love how out-of-touch HN comments can be sometimes, it’s a breath of fresh air for me. Outlook the mail software (as opposed to the coat of paint applied to hotmail) is used by almost the entire Fortune 500, basically every government at every size, and a decent chunk of the Windows-using western world (which is most of it).

If you’re wondering why you have trouble selling to enterprise and government clients, this type of gap in perception is a big part of it.

The point is mail is run through Microsoft Exchange, not the client software.

Given how wrong the rest of the comment was, confusing mail client and mail server seemed like small potatoes. Especially when it can be argued that mail flows through the client if you squint a bit.

What was wrong with the rest of the comment?

Many enterprises have moved to Microsoft 365. Which uses Outlook.com and not Exchange.

The mail service in Microsoft 365 is called Exchange Online and it‘s based on Microsoft Exchange. Outlook.com (formerly) seems to have been been moved to the same infrastructure as Microsoft 365.

It's hosted (at Microsoft), but it's still Exchange. Though maybe there's a lower-end, non-Exchange option?

In any case, the article is a little confused.

> If you’re wondering why you have trouble selling to enterprise and government clients, this type of gap in perception is a big part of it.

Most HN founders aren't on YC anymore aside from their 1 free Show HN post and the account you make for the YC app.

Speaking from experience, they have much healthier discussions on Bookface, Twitter, and in person.

You're basically stuck talking with salty burnouts on here (to paraphrase a YC friend of mine who has publicly hated on HN)

I think it was more a comment on the fact that Outlook is the mail client, not the MTA.

Gee, I wonder why F500s are getting ransomwared left and right :)

I get that giving your drones software that is widely used is the easiest path but ActiveDirectory, Outlook and Windows are basically three of the horsemen of insecure corporate networks. The fourth is probably adware disguised as virus scanners.

Ok I'll bite -- what are they supposed to use instead of ActiveDirectory? We going back to Netware?

Oh you don't want them to use Windows at all? So what, F500s are gonna run on Ubuntu and OpenLDAP? That should be smooth as butter and more secure out of the box /s

Thanks for explaining!

> Imagine if that happened during an actual crisis.

I’m ironically imagining our enemies are coming up with some heinous reply-all scheme that will bring us to our knees.

It's called "Social Media".

Some PsyOps person on here had a term for this, slowing your enemy down by getting them bogged down in unimportant details, but I've forgotten what it was...

Thus one who is skillful at keeping the enemy on the move maintains deceitful appearances, according to which the enemy will act. He sacrifices something, that the enemy may snatch at it.

By holding out baits, he keeps him on the march; then with a body of picked men he lies in wait for him.

- Some famous Chinese PsyOps dude

https://ronaldthomaswest.com/2017/08/21/the-gentlepersons-gu...

Topic dilution is the term you're looking for.

Sounds like a cross between bikeshedding and nerd sniping.

>It's 2023 and someone is still writing that?

Journalism isn't the profession it once was (if it ever was) from major corporations. They're just writing information they are fed by a source and spruce it up a little for views, maybe. There's no investigation going on anymore, there's no source checking generally, at least not like there used to be. The current business model doesn't allow for that in large organizations.

It's Politico, not an IT news source. I think someone on deadline potentially confusing Microsoft's client software and it's MTA is forgivable, particularly given the ever-shifting landscape of MS' branding in this area

Honestly, that's fair. I just haven't read something like that in a major publication in a long time.

I mean sure, Outlook was only released ~26 years ago, I'm sure they'll catch up eventually.

Why would you give them a break if this should've been tested, found, and fixed, ages ago?

When did this project start?

Reply-All cascades are classic entertainment in government work.

When I was in the Air Force some poor Warrant Officer started a Reply-All clusterfuck when he replied to a base-wide email about issued Android phones.

After about 30 Reply-Alls some very senior officer replied (to all) something to the effect of "Nobody else Reply, or your ass is grass" to which some joker from Special Operations replied (all) "Copy, Sir".

Some of the most fun you can have in the public sector.

We had this in college where someone sent "M to mute" and then thousands of people started sending emails with "M" in the body. Nobody knows how many of them were being sarcastic, but it became a meme that we found ways to insert into all sorts of unwanted group communication channels.

When the friends groupchat becomes overly active about a topic only few are interested in (not including me), I joke and reply UNSUBSCRIBE. I think I'm the only one to understand, and definitely the only one to laugh

These are just as fun at large corporations. A bunch of people you've never met, all very mad at each other. (I always join in on these. I simply cannot resist. One time, someone emailed me privately to say that I should know better. Oh, I do. I just want to watch the email server burn.)

I feel like the sender is more at fault in these cases. BCC exists for a reason.

I really don't know why it's so hard to get basic competency in public sector to use bcc or restrict send access to large groups. That said, I enjoyed the drama when an email devolved into requests to stop replying all and requests to be taken off the list.

If you would like more info on that topic... Was on HN back in early summer. https://news.ycombinator.com/item?id=36620608

"A recent international research study allows us to quantify the difference between the broad population and the tech elite. The data was collected from 2011–2015 in 33 countries and was published in 2016 by the OECD (the Organisation for Economic Co-operation and Development, a club of industrialized countries).

In total, 215,942 people were tested, with at least 5,000 participants in most countries. The research aimed to test the skills of people aged 16–65."

Summary Page: https://www.oecd-ilibrary.org/education/skills-matter_978926...

Executive Summary: https://www.oecd-ilibrary.org/education/skills-matter/execut...

Key Adult Proficiency: https://www.oecd-ilibrary.org/education/skills-matter/adults...

Nielsen / Normal (focusing on computers): https://www.nngroup.com/articles/computer-skill-levels/

USAF doesn't have warrant officers (used to, but the last ones retired last century), but yeah I saw a couple of these in my time, too. Not a lot of fun.

Never said I was USAF.

These days you can just mute the email in Outlook.

At my Amazon peers...

"Wallet"

Context:

At amazon in 2010, it started with a meeting invite that was accidentally sent to everyone. The meeting was for the Amazon Wallet team for whatever work they were doing at the time. A reply-all storm ensued, and despite several people asking to stop hitting the reply-all button, it continued for several days. The management had to chime in and sometimes threatened people with punishment if they continued participating to the reply-all storm.

If you ignored the conversation you may not have seen it, but most reply-all storm ended up with someone saying something the line of "we don't care about your wallet".

Anyway, asking people to stop hitting the reply-all button is far from being the latest reply-all on these kind of things...

Is this a copypasta? If not, it should be.

Here is a story of a similar form:

https://devblogs.microsoft.com/oldnewthing/20111014-00/?p=93...

It’s a pretty predictable fact pattern.

It's not a copypasta; I was there for Reply All 2019.

I finally got around to watching The West Wing. There's a lighthearted small subplot of exactly this in S01E19.

By the way, it is so refreshing nowadays to watch a show about extremely competent imperfect people that legitimately strive to do as much good as they can, as they see it. Likewise with Sorkin's writing, while unrealistically witty and fast, it is a delight watching someone so obviously talented give their all to their work. Granted, as in other Sorkin writing, it does sound a bit like all the characters are just versions of him with different voices. But it seems to me like an intentional stylistic choice, a somewhat heightened reality.

The movie Malice is one of my all-time favorites. 1993, average reviews. Total random movie. Recently I paid attention to the credits: written by Aaron Sorkin. It all makes sense now. It's been ignored for whatever reason but it's amazing.

I call this competence porn - a brief escape to a world where leaders are both competent and benign.

My other favorite is when corporate comms or HR sends out some calendar invite to a massive number of people, does not use bcc, and then the server crashes when folks go in to the accepted invite on their calendar and you get thousands of calls to 'check availability' flood in.

Have seen that bring the Exchange Server to its knees a few times :)

Or the out of office auto-replies. Heh.

This problem is relatively easy to solve with a Global rule:

- TO: [All]

- Subject: ^(?i)(re|fw):\s

- Action: Drop.

Heck, you could go one step further and block all TO: [ALL] entirely since they should BCC.

Way back in the day when I was doing a lot of Exchange consulting work, we would recommend removing permission to send to all except for one or two specific accounts. This was to avoid reply-all meltdowns, for security (viruses couldn't send to all), and also from an HR / internal communication perspective. We would recommend something similar for department-level groups, but often with a bit more freedom.

The latter is best. I think another sensible action is to remove “all” from the reply. If you really want to send to all, you have to enter it yourself.

If you go this route (which isn't great), you probably want to include "Fwd" as well.

I've seen this happen a few times, usually someone accidentally spams their entire address book, or even better a bunch of listserv addresses. Then you get the "reply all" messages to "take me off this list" or "unsubscribe" which just adds fuel to the fire.

The last storm I was in had people replying all to tell people to not reply all, followed by people replying all to shame the people who replied telling people not to reply all.

Followed by others replying all asking how to mute the conversation, and others replying all with instructions. Multiple times. Which prompted more people to reply all commenting on the increasing recursion of the absurdity of the whole thing.

So you'd get fed up and mute the chain and think you'd think you'd killed the process, but any time someone changed the email subject line and replied all again it came back like a bad penny. Which caused more people to angrily reply all with vulgarities, which caused supervisors to reply all that their behavior was being noted and disciplinary meetings would be had.

And then one day, someone replied-all to one of the child email chains with the entire script of The Bee Movie. Which launched more of all of the above. And did the same thing to every other child email chain, sporadically, for a few months. I think it took over half a year for the last reply-all to quiet down.

I think at least one of those chains lives on for posterity in one of my folders, and now you tempt me...

One of the privileges of previously being an Exchange admin was being able to send the "Do not reply all" message and then setting up a rule on the server to kill all further messages.

But it's also why I had a script to lock down any distribution list with more than 100 members to authorized senders only.

Thank god for Transport Rules to just silently drop emails with matching subject for a few days.

This happened today at work. Like just ignore the email...

But that's no fun! ( •̀ᴗ•́ )و

Back in the days before ransomware and RATs and APTs and ubiquitous cloud-hosted corporate email, when viruses were accidentally introduced rather than spear-phished, state/criminal-gang-sponsored existential threats to a company, I had the idea (without the ability to manifest it) of a virus that would infect an MS Exchange instance and set everyone's Out of Office on and all incoming email to be forwarded to every other mailbox in the company.

Not sure if it is or was feasible, but I still like the simplicity of an automation-stack doing the really hard work of exponential amplification.

Amusing, almost 30 years later and this same problem still exists: https://www.linkedin.com/pulse/surviving-bedlam3-email-storm...

It's 2023, we have systems processing trillions of transactions per minute, and still a few thousand emails can "melt" an email server? That's testament to how little consideration the mail stack gets.

Well, kinda. Due to features like BCC headers, each recipient is supposed to get their own unique copy of every email, so email servers can end up spending a lot of time writing tens of thousands of copies of every mail sent to the list to disk. And then of course they have to sync that to all of the redundant servers, to support HA.

Symbolic links and deltas are a thing. None of this is hard from a technology standpoint.

Exchange just isn't the best designed or most modern software.

I don’t entirely disagree, but while deltas can be small emails are usually pretty small themselves. It might be a single 4k write to each mailbox either way. And remember that appending to a file requires the OS to do a read plus a write, while adding a file to a directory requires a write plus an update to the directory itself (another read+write at minimum). Both require allocating sectors, which means updating the freelist(s), Of course there are filesystems like ZFS that can batch up all of those writes so that they become linear writes instead of random writes, but that’s not available if you‘re running Exchange.

I have no doubt that email servers could be better optimized, but only because I have that optimistic belief about all software as a general rule. (The probability that we have already found the most optimal way to write any non–trivial program you happen to examine is pretty low.) On the other hand, I doubt that Exchange has made _no_ progress at all over the decades.

The fact that their email server fell over from the load probably has more to do with poor choice of hardware than any software flaw. Their Exchange server apparently fell over while sending email to “thousands” of people, while the Bedlam DL3 incident at Microsoft involved a mailing list with 13k people on it. In 1997. Maybe that order of magnitude difference means that the Senate needs to buy an NVME disk. (Of course that ignores the fact that asking thousands of people to send you their location via email is pure stupidity; this is the Senate we’re talking about.)

Surely there's a better way of getting everyone's location in an emergency. Although, long/lat aren't easy enough to say, type or find.

Seems trivially easy to solve since all the users are already carrying tracking devices with GPS.

Presumably they are more interested in information like which room someone is in than their position on Earth. I doubt that indoors the location accuracy of phone GPS would be accurate enough to be useful, particularly in buildings where many locations are designed to be secure both from physical attacks and espionage (probably a lot of thick metal plates in random walls). And even without interference issues, GPS elevation accuracy is pretty bad. Finally, you're only tracking the location of the person's phone, which in an emergency could easily get left behind somewhere.

None of this is to say an email chain is the optimal solution though.

If there's a cheap, simple way to implement it - I'd quite like to know.

"You guys all have phones right?"

The gov could write an app, installed on every senator's phone, that tracks their location as needed.

Perhaps the senators might be against such invasive surveillance? Then I'd hope their voting record on such bills matches their privacy concerns when they're the ones affected.

That gives location. Not status. So it tells you whether they are in the building, but not if they are fine, injured, incapacitated or unconscious.

Moreover, lets say the senators have a mixed record on massive invasion of privacy. You need a system that covers everyone

MS did it first: https://techcommunity.microsoft.com/t5/exchange-team-blog/me...

Remember the days before mailing list distributers filtered out of office replies? It was fairly common to open your mail in the morning and have to delete several hunderds of these auto replies and dozens of replies from people complaining about it adding to the bulk.

This happened to us several years ago, a basic reply-all storm across probably 30k employees. The icing on the cake though - when the storm finally calmed down a few hours later, the original sender then tried to recall the message. The recall notification went out to everyone and sparked it back up again.

Got caught in 2 different bad reply all storms at Accenture a few years ago.

The temptation to respond and mock / yell at the people who keep replying is strong.

Feel like it should be a fireable offense at a place presumably hiring for critical thinking and situational awareness.

And here I thought the oxy-moronic reply-all pavlovian response had been trained out of humans. I've been fortunate enough to work on teams where I haven't witnessed this in at least a decade. Of course its in our technically inept government where this happens.

Of all things happening in a West Wing episode (although to the White House, not the Senate) to still be topical 25 years after airing, this would not have been high on my list. I'm not sure to consider myself overly optimistic or overly naive...

Great way to keep busy on anything but passing any sort of funding bill!

In college, I came up with a one-liner to send the entire MAN to every email address. The server outage disrupted classes for a week.

I can't come up with anyone who wasn't >irritated with me.

Same thing happened at Microsoft, and Facebook. I'm sure it's happened anywhere with large email lists. Servers should probably just reject replies to lists large lists.

https://en.wikipedia.org/wiki/Email_storm

For Amazonians - Wallet?

Details on the Wallet thing: https://news.ycombinator.com/item?id=37336088

Related recent thread with lots of reply-all anecdotes:

* @gmail.com (xkcd)

https://news.ycombinator.com/item?id=37333848

Terminate everyone who pressed reply-to-all as they can't be trusted to follow simple directions and they shouldn't have a security clearance either.

Reply on news.ycombinator.com