Companies may be losing millions due to emails buried in collapsed Gmail threads | Hacker News Reader

Companies may be losing millions due to emails buried in collapsed Gmail threads

thehftguy.com

244 pointseaguyhn183 comments

I worked at a place that had many customers who used a major free email provider that also ran an ad network (but was not Google or Microsoft) that kept deciding that most mail we sent to our customers was spam: receipts, installation instructions, responses from support, and notices that subscriptions were going to re-bill were all blocked.

We'd contact them, they would fix things, and it would work for a while...but invariably it would go back to classifying it all as spam, and we'd get customers calling to complain about not getting instructions or support responses.

Finally, our guy who managed our ad spending called up our rep in ad sales at the email company, and asked in rather forceful terms just why we should continue spending $large_number/month on ads there to acquire new customers, when we were then going to be blocked from emailing those customers?

The ad rep conference called in their head of IT, who conference called in their engineer in charge of spam filtering, and had him right there add us to a whitelist that would prevent anything from us being classified as spam no matter what the rest of their spam system decided. As far as I know we never had another email go into the spam filter.

If we hadn't been spending a large amount on ads with them, I doubt the problem would ever have been resolved.

If you sell a software product to end users, and have non-marketing messages that you need users to see such as responses to support requests or reminders that subscriptions are going to automatically re-bill, I'd recommend including some kind of messaging system specifically for this in the product itself. Still send them as email, but also send them via this in-product channel, or at least send notices that there is a message with links to a way to view it on the web. Email only is simply not reliable enough.

> Email only is simply not reliable enough.

That is because implementations, especially gmail, ignore the specifications: (EDIT) there are specific error codes and messages should be sent back to the originator. The reasons to ignore it is, in theory, not to give spammers ideas how to avoid it.

Email is incredibly reliable, it can tolerate outages, configuration errors, delays, etc, by specification, but ever since "modern" spam filtering, gmail (and those who follow their practices) it became a nightmare, because everyone ignores how it _should_ work. It was done in a time when connections were slow and prone to errors; it had to be flexible.

Yes, I know spam was bad I was there. But I'd rather have spam, than actually important mails getting dropped because of false positive, paranoid spamfilters.

You’ve just explained why it’s become unreliable. It might have been great once, but fixing it when it’s broken is a complete nightmare now.

When there’s too much spam, your important mails are effectively dropped as they’re buried in the noise. Modern spam filtering is what saved email. Without it, you’d never see those important messages, because new spam would come in faster than you could delete it.

I don't have any content-based spam filtering (but I do use the spamhaus sbl-xbl list).

It doesn't take very long to manually delete spam from the inbox. And if your spam filter is putting legitimate emails in your spam folder, you need to manually filter them anyway.

At the peak of my spam experience, I was getting hundreds a day. That was ages ago when the internet was much slower and more expensive than it is today, and bad people online were much less sophisticated. I wouldn’t be surprised if a totally open email system today generated enough spam that even deleting one message per second you’d fall behind.

I haven’t checked my spam folder on a routine basis in many years.

What I'm saying is that "modern spam filtering" (by which I assume we mean GMail-style highly-aggressive spam filtering that puts lots of legitimate email in the spam folder) is not what saved email. Rudimentary spam filtering saved email. There's no need to ever have a legitimate email go to the spam folder.

Spam filtering is an arms race. Spammers know how to download all the existing spam filtering software and tweak their emails to make sure it gets past it.

My experience matches jstanley's: with basic ISP filtering and disabling as much filtering as they let me disable my ISP reports blocking 181 messages in the past month mostly via some list (they don't say which one). I see a few spam messages per day and generally get half a dozen or more copies the first time I see a new one, conveniently making them easy to delete (and very few have confusing subjects). I'm fairly sure I see less than half as many as the ISP blocks. This is with an address that has been public for over a decade. It doesn't seem like much of an arms race to me and it is a trivial effort to delete them.

The only things I can think of that I do differently than many people that might make a difference is that I have my mail client set up to not show remote images ever and I never dirctly click any link from email no matter the source. Possibly things might be much worse otherwise, but from my perspective it doesn't seem like spam is a huge issue these days with basic filtering.

You're still benefitting from everybody else's more sophisticated spam filtering. The fact that most users have good filtering means that the cost/benefit ratio of sending spam is much worse than it otherwise would be. This naturally decreases the activity significantly. Saying that you get by with just basic blocking is the e-mail equivalent of saying that you don't need a measles vaccine. It may be true but it doesn't tell us much about what the world would look like without those measures.

Where is the evidence that this is true? It doesn't seem that likely to me that so few people are not using gmail-style spam filtering that spammers couldn't figure out how to target those of us who aren't (the filtering that I could have used but turned off is more random than helpful and I would guess this is the case on many non-Google mail providers). Even with Google running a huge portion of all email there are still a huge number of people with email not provided by Google.

An alternative narrative is that previously email servers were supposed to accept almost anything from everyone and spammers took advantage of this. From this perspective it is the relatively recent requirement of sender verification that made the key difference and allows Spamhaus to work (obviously they need to do something to determine who to block). This perspective seems to match what I am seeing.

Rudimentary spam filtering saved email long ago when spams were focused on a few niches like pharmaceutical or Nigerian scam.

It wouldn't be efficient today with spam being more more diverse and often hardly distinguishable from legitimate emails. A lot of spam nowadays is like forced advertising, by opposition to plain scam and dodgy stuff.

The definition of spam changed. Originally, it was scattershot bulk email, where the sender would just send to any address they had ever acquired for any reason.

Now spam means "anything the user doesn't want to receive". Even if the user previously deliberately created a relationship with the sender. Gmail successfully trained users into clicking the spam button for anything such. People just click Gmail's magic make-it-go-away button for things as legitimate as a mailing list they deliberately signed up for and are simply too lazy to properly unsubscribe.

Legitimacy isn't a binary either-or, it's a continuum. You bought from a merchant ten years ago and they email you again, is that spam? How about the company that acquired that merchant which is now trying to sell you a completely different line of business? You donated to a politician and now their successor in the same party emails you, is that spam? That's what modern spam filtering deals with.

'Now spam means "anything the user doesn't want to receive".'

Spam means anything from someone who is indifferent to whether the recipient wants to receive it and doesn't get "sufficient" permission. I think this is consistent with the original type of spam, even if there is no bright line delineating it.

If you're implying that anything with a working unsubscribe link shouldn't be considered spam, then I think this definitional issue relates to the same controversies about what constitutes consent that have been discussed a lot recently.

I mean, I agree with your last paragraph, so far as that goes, but I would suggest rather than the definition of spam changing, ordinary businesses got spammier/sleazier.

> It doesn't take very long to manually delete spam from the inbox.

I have tuned the spam filters for a fairly major email provider for a while and while you are right for a lot of people, there's a significant fraction of them that receives hundreds or even thousands of spam emails per day. For them, it is not practical - and everyone needs to be able to use it.

I have subscribed to more email lists these days than ever before. Why? Because trust in the system has increased. I know that I can easily subscribe/unsubscribe and that decision will be respected (or punished if ignored).

So while gmail has made life harder for people who want to email me, it have made life much easier for me as an email user - which makes me more willing to give out my email.

SMS

Notifications

They all go through some provider who can block you.

It’s a rare product that people check on their own without notifications!

Any ideas how to reach people without gatekeepers? I have just one way: a desktop app or mobile app that is allowed to periodically wake up and poll a server.

A printed invoice in an envelope and sent via Post mail usually still works.

Then we have the problem of a service that doesn't sort out spam. I often get enough junk mail that it's hard to fit everything in my mailbox. Important mail ends up shoved between pages of catalogs I'm about to throw out or mixed in with an ever-growing stack of junk that I'll sort through later when I find the time.

It's a DOS attack on your mailbox. The SNR is approaching 0.

You’re kidding, right? The Postal Service is the ultimate gatekeeper. You have only a companies that would make the last mile delivery.

> Any ideas how to reach people without gatekeepers? I have just one way: a desktop app or mobile app that is allowed to periodically wake up and poll a server.

If you're asking "anybody today" then yes, that's probably one of the only ways.

But maybe tomorrow might be better.

On the horizon (where it may stay - it's been slow in coming) is the idea of "distributed and serverless" internet. Also known as "peer-to-peer".

You can think of it as everyone (well, their device) being their own server - kinda like the internet was originally meant to be (before vast sums of money got involved).

Through "magic" - each person that is on the network can browse, send, receive, post, etc - to others without needing much of anything else - just the pipe (not really any way around that kind of "gatekeeper" unless you have a mesh network of some sort - and even there, long-haul can be tough or impossible). A well known example:

https://ipfs.io/

...there are others out there as well; just google around for "peer-to-peer web" and similar terms.

Funny that you mention that. One of my company’s initiatives is building social networking software to run on local networks and mesh networks. I also want to see a future where cellphone signals, energy generation and so on don’t go through “last mile” gatekeepers but those are just one of many commodity choices to connect to “the grid” / “the cloud” / “the mesh” whatever it is called.

I think that is super important given what’s going on:

https://qbix.com/blog/2019/03/08/how-qbix-platform-can-chang...

We have started to work on captive portals so your device can surf from wifi to wifi just like a cellphone does with towers, and check in wherever you visit without you doing anything (with your prior permission, of course)

https://qbix.com/blog/2017/12/18/power-to-the-people/

IPFS is great, as is BitTorrent. I have met a lot of the people running these projects, like Tom Berners-Lee and the guys from Solid (back then), David Irvine and the guys from MaidSAFE, and Petar Maymounkov from NYU who invented Kademlia DHT.

I think there are several things that are needed:

  NAT hole punching
  End to end encryption
  Kademlia or other DHT
  Small group consensus
  Merkle DAG with validators

All these can be incrementally added to an existing, web-based solution. The problem with BlockStack and MaidSAFE is that they require you to download a new browser that only works on the desktop. Instead of trying to build on the existing Web.

For me it’s not even so much about privacy as being permissionless. No gatekeepers.

Whether it’s your cellphone company or your cable internet ISP or the Google Maps API or Amazon Web Services, they require you to pay them for their closed-source infrastructure and data.

The infrastructure should be a mesh and the data should be open source. The challenge is simply in the software to coordinate the mesh participants, and for the data it’s software to enable the right rules to maximize the chances of data being correct.

There is tons, tons to do and frankly most VCs and investors don’t get it. The payoff is huge for humanity as a whole, and can unleash even more innovation than the Web did after it disrupted AOL etc. But they don’t capture even 10% of the whole value. It’s free to the world.

So we’re chronically underfunded but we have already put together so much. And now we are making money the old fashioned way: earning it from users and clients hahaha. The Basecamp guys would be proud.

Blockstack Engineer here. You do not need to download a browser to use Blockstack. We also have a working mobile version, and multiple apps built on the Blockstack API that have their own ios and android apps. You can check out Blockstack without downloading the browser here: https://browser.blockstack.org

If it is your first time, you will be asked to create an identity on Blockstack: https://docs.blockstack.org/core/naming/introduction.html

Each instance of the "browser" whether on desktop or phone, requires a unique password, associated with the id. The id is stored on the Blockchain, the data is stored in your own gaia hub which you can host wherever you like: https://docs.blockstack.org/storage/overview.html

In regards to your comment about "Kademlia or other DHT": Blockstack previously used Kademlia, but iterated on an improvement to Kademlia by creating Atlas which is what we use at Blockstack. More here: https://github.com/blockstack/atlas

and the specific reasons about why Atlas was built in lieu of Kademlia here: https://blog.blockstack.org/blockstack-core-v0-14-release/

Together, Atlas, BNS, and gaia create the foundation upon which the Blockstack API is built on.

You can build applications using the Blockstack API here: https://docs.blockstack.org/develop/zero_to_dapp_1.html

If you want to explore the ecosystem of existing decentralised applications on Blockstack and other decentralised solutions, you can see that here: https://app.co/

You can read more about how identity, Atlas, and user owned storage works here: https://blockstack.org/whitepaper.pdf and maybe poke around here a bit: https://core.blockstack.org/

Oh yes, also you can download the browser on your desktop if you like.

Hey, are you around in NYC? Would love to grab coffee with you and discuss archtecture together. I am curious how Kademlia and DHT turned out to be unreliable in practice and the lessons you learned for why what you use now is better

Can you drop me a line at username greg with the domain qbix.com ?

Absolutely. We will be in touch shortly. In the mean time, you can check out our forum: https://forum.blockstack.org/ or hit is up on our slack.

Cool — what to look out for, to make sure I don’t miss it?

Hey, didn’t get any email from you

Since you are name dropping projects:

Tox, a P2P Skype alternative.

Unfortunately it is not well known outside of tinfoil hat community.

> wake up and poll a server

Here the AppStore is the gatekeeper. OP is probably right, your website is the most reliable place to communicate with your customers.

I have a Gmail nightmare story from a couple of years ago.

The company I work for was sending confirmation emails doing everything properly: separate IP range mx boxes, warmed up, DMARC, SPF, DKIM, all the bells and whistles. Nothing else was allowed to go through them.

Suddenly people weren't getting those confirmation letters. It turned out that soon after Gmail introduced the "Promotions" tab, a silent, new feature was added: anything from noreply@ was put in there.

We tried going through the official channels - once we got to the end of the tunnel, it told us "we'll get back to you in 2 weeks". That's when we started digging into internal google connections across the company, and thank god, we found someone who know someone, and in was sorted within a couple of hours. (It might have helped that the company spends a lot of money on google ads.)

Gmail is lovely.

You should never have been sending mail from noreply@ in the first place. Every message needs to be send from a real mailbox that can receive a reply that will be read by a person. If the message isn’t important enough for that, then it is spam and Gmail was doing the correct thing with it.

I don't believe your approach is correct, for multiple reasons.

There are valid scenarios for one way communication; those emails are only verifications of transaction that can't be simply altered via responding to a mail.

Email has the notion of Reply-to header, although this case, that was not populated.

Making a decision solely noreply@ is why the life of Mr. Null is quite hard.[^1]

No, gmail is not right about this.

[^1] https://www.wired.com/2015/11/null/

It doesn't matter if the transaction can't be altered if something went wrong. The customer can still have a complaint to register, like "I am not your customer, please stop sending me his receipts".

I have to disagree with you. If you're sending a message to somebody, no matter what the medium, it's the right thing to do to give them a way to respond. Emails are a two-party transaction, not a facility for unidirectional spam.

A valid return address is not the only way to respond. There could be a link to a support page in the mail, a telephone number or even a mailto: link.

There could be, but frequently is not.

gsuite-noreply@google.com sends me invoices.

What's your point?

You can respond by looking up the company and contacting them...

Sorry, this ship sailed about 25 years ago. These days, emails are primarily a facility for notifications and other unidirectional messaging. Some of it is not spam. A lot of it is. I delete rough 20 emails every morning, unread.

Postcards have a long history of disagreement with you.

You generally don't receive postcards from people you don't know how to get in touch with.

Do you even feel this way about emails used for a password-reset transaction?

Absolutely. Why wouldn't I? Let's say I didn't request it. Wouldn't you want a way to reach out to support to let them know that someone might be trying to get into your account?

If you want your support staff responding to user's auto-responders, newsletters and spam, then use support@, but if you want your staff to get work done, then you use a noreply@ and have your support requests come in some other way.

You seem adamant that users and their ISPs should deal with the spam problem without any speed bumps for you, but unwilling to accept the equal spam problem of sifting legitimate support problems from other replies.

I understand the frustration of seeing our simple and low effort mechanisms decay over the past 30 years. But I also appreciate that there might just be a need for a more symmetric social contract for engaging in the use of communications systems and other common infrastructure.

I don't consider receipts and other noreply content from the places I frequent to be spam. I want those messages. If my ISP or service provider auto-spammed them I would be very annoyed. Just because I want their receipts doesn't mean I expect to get a live human when I reply. I'll just call them or email their support line.

Hmm, I didn't mean that all notifications are spam.

But, the task recipients face for sifting and sorting their inbox into different buckets is essentially the same faced by a notification sender who accepts feedback instead of hiding behind a no-reply address. Lacking real message authentication and authorization tools, we have to look at a mixture of content and metadata patterns to try to sort it out.

In this every gray world, I can imagine reciprocity being a useful measure...

Is that a 'should' as in best practice, or a 'should' in terms of a RFC for a mailing standard?

The reason I ask is because I'm pretty sure lot's of sites (invoices/receipts/newsletters) send out stuff via a noreply intentionally.

>a silent, new feature was added: anything from noreply@ was put in there.

This is my frustration with Google and their suddenly deciding this is how things should be done. For instance I use my camera to capture receipts, QR codes I may want later and a dozen other things. Google is of the opinion this wastes space so I get regular "Clear up the clutter!" cards. One man's clutter is another man's system of record for important stuff. Stop imposing your opinion on how I should use your product, Google.

I believe you can turn those off.

Ah after some more googling it looks like that's controlled by the "suggested archive" option under google photos. I'll try it and see.

Holy shit, Google editorializes the content of your own personal photos? I'm obviously way out the loop with my gapps-less LineageOS phone, because the Overton window has clearly moved a long way.

Article admits it isn't a bug then continues calling it a "bug." Also the $187M figure is just pulled out of the air.

Essentially Gmail (and other mail services) put messages from the same sender with the same subject into a Conversation View. Users are confused by it, and clicking the oldest password reset link (expired) rather than the newest.

Claims that users being unable to reset their passwords costs Expedia $187M.

Oh..and the clickbaity headline got me again.. All while I was looking for how some hacker exploited it and got Expedia's business

LOL, I guess it's anything for clicks these days...

Nah, the author does have a point. There's something fundamentally wrong with how Gmail collapses messages. I have email notifications from a certain web store when some of the items I might be interested in come in stock/for preorder. I always read them. Years after I started using it, I discovered that I actually miss reading about 30% of these messages. How does this happen?

Well, it turns out that Google randomly groups some of these messages in a single thread. And yet this thread looks identical to a single message. So I click on the thread, read the first message, see the email footer and go back to Inbox to read the next one. This marks the entire thread read even though I didn't actually read the subsequent messages. So I never come around to read them.

I completely believe that most users will not read any email in the thread beyond the first one, because there's absolutely no indication there's more stuff to read.

Also, long exchanges with someone who uses Outlook as a client are incredibly painful.

Does it not collapse them? I have a few people I used to talk with that had the included reply but GMAIL just replaced it with [...] that I could click to expand.

I came to say the same thing, but you said it first and probably better than I.

When gmail was invite-only, the threaded collapsing feature was amazing and perfect for what I got my invite to do (roleplaying). Since then after innumerable redesigns and behavior changes, I don't even remember what the old behavior or look really was like, except I liked it and had fewer complaints. These days I'm annoyed that gmail doesn't collapse messages enough, i.e. the "random" threading or not threading of emails that seem like they ought to be part of the same thread (some seem only different by time). I get annoyed with their UI on multi-party emails. And you have a point about marking the whole thread as read when you didn't even scroll the wheel once, though you do have a number indication before drilling into it of how many message are there. Make the number bigger? If you try to reply it's also obvious there are more messages. At least it knows you read earlier messages in the thread and collapses them by default.

There's still a lot of improvements for mail UX. I don't expect google will be the one to make them, but maybe they'll copy the other people who do.

> Essentially Gmail (and other mail services) put messages from the same sender with the same subject into a Conversation View.

IMO, gmail is in the wrong here. If the message id is not in the References or the In-Reply-To header, it's a new conversation.

They should just make that a configuration option. I prefer it works like Thunderbird, where no new mail is collapsed (but I can thread them if I wish).

EDIT: Hmmm. It appears that I CAN change that.

How?

I went to settings (gear thingy->settings) and set "conversation view" to "off."

This is kind of a silly argument to make. How does an end user have any way to know whether a particular behavior is a bug or intentional when the behavior is actively bad?

Recurring messages having the same title is an absurdly common pattern. Almost every service that has ever sent me a password reset email has used the same exact subject. Same goes for things like shipment notifications, order confirmations, purchase confirmations from PayPal, etc. So given this, it's kind of absurd that a MUA would be designed in a way that buries all those emails, isn't it?

Let's assume that it really is intentional that gmail buries all those messages, on some presumption that they aren't important or on a 'well fuck you, change it then' basis. There definitely are services out there that append random numbers to the end of automated emails. I always assumed this was to make it easy to sort threads by unique id, i.e. customer service systems - so maybe they've been applying an undocumented Gmail Best Practice this whole time. Assuming that this is a good feature implemented correctly, why does unread/read status not work right? If I click into a 50 email long unread thread and then click back why is it INSTANTLY marked read? How is that a useful behavior that would seem intended to anyone? Non-threaded views like in Outlook do not work this way. The common 'reply up top, history at bottom' email formatting also avoids this problem, which you'd think the gmail frontend designers would be aware of.

I would argue that both behaviors are either a bug or user-hostile design. Calling it a bug is generous to the designers because it assumes goodwill and just views it as an oversight or error in a very complex system. I'd personally be inclined to call it bad design, because Gmail is full of bad design, but there's nothing weird or bad about a user calling it a bug!

A $187M figure is super realistic to me given how often I see this particular problem affect me. It literally happens daily. Naturally, I learned years ago that gmail does this and got used to having to dig through my email history to find out where a notification went, but it's still a bad behavior and it still catches me unaware sometimes. I've missed important emails this way.

The difficulty of maintaining Inbox Zero in 2019 also combines poorly with these behaviors - when a notification for password reset or whatever gets threaded in to an email with an old date on it, it can make it harder for your brain to process what just happened.

This combination of behaviors has long been the key to me hating gmail. I can't begin to estimate the number of messages I've missed because of how gmail handles threads.

I had the exact same problem setting up my Dad and My sisters new ipads at Christmas.

Author here. Just wanted to say that the metrics are not pulled out of thin air. I prefer to refrain from discussing the internal accounting of firms I worked for.

Many of my customers use Gmail (mostly as part of a company subscription to Apps) and I have frequent problems with their mail getting lost, classified as spam, rejected, filed somewhere, or just difficult to see.

I wonder at which point people will start realizing that Gmail is not all it's cracked up to be. Apart from the privacy issues (you basically have to assume that Google is reading all your mail and mining data from it), Gmail treats your mail as their mail: they will do anything they like with it, including hiding it from you.

If that sounds like a rant, it is — I am worried about the increasingly centralized nature of E-mail.

They claim that they don't use your emails on gsuite the same way as they use it on gmail proper, but I find it very difficult to trust them on that.

To betray their paying customers’ trust on the pennies they would make would be a truly boneheaded move.

Not everything is about money, you know?

Your post implies that companies (or even just Google in particular) don't make a habit of making truly boneheaded moves. I think that's a pretty difficult position to back up.

Didn't Google do exactly this with school accounts a while ago?

Researched briefly and this was the best I found but I have memories of something even more juicy: https://www.theguardian.com/technology/2015/dec/02/google-ef...

According to a 2017 Reuters report (most recent I found), G-suite gets them revenue of about 1.3B/year. Advertising revenue is more than 80B/year.

It might be scummy, and certainly risky, but you can hardly say it would be boneheaded to risk damaging a $1B segment to improve an $80B segment.

Never forget: Google is an advertising company. They're not a hosting company, or a SAAS company, or a phone/laptop company, and certainly not a browser company or a mobile OS company. They may dip their toes in lots of areas, but the core is solidly about advertising.

Conversely, foregoing additional ad revenue and other benefits of using emails from the relatively few number of G-Suite accounts likely has an incredibly small direct impact on the bottom line. I can't speak for any internal data Google has, but my guess would be that individuals using G-Suite accounts are more likely to use Google services in their personal lives. Any lost benefits or ad revenue would be easy to write off as a negligible loss leader.

Heck, with how common G-Suite is used in schools, getting kids used to Gmail early would be a powerful argument on its own. Though writing it out like that kind of feels a bit dirty.

There are plenty of companies that manage to make money from a diverse portfolio. If Google wants to be successful in enterprise offerings they need to set up business units with P&L responsibilities so that they don't conflict with the rest of the business. Or they should spin it out as a separate company. There are lots of ways to solve this problem.

Speaking as a long-time G-Suite user the offering currently feels a little lost. It's a bit disappointing considering the initial promise of the product. Aside from personal inconvenience, letting it die would be damaging to Google's ability to sell into enterprise markets.

But how much is the data mining worth?

> If that sounds like a rant

Not to me, and probably not to anyone else that has to administrate GSuite. I am continually astounded at how lacking in features and management capabilities corporate gmail accounts are.

Its a discount office 365 in every way - lower price, lower features, lower usability.

It sure seems like google is itching to discontinue.

I've implemented G Suite for over a dozen companies. I have no idea what you're talking about.

I recently applied to be a volunteer first aider for St John's Ambulance, who are fairly prominent in the UK. It was 2 weeks of no response to my application, until I happened to check my gmail spam folder to discover that SJA had invited me to a selection evening like the day after I'd applied!

Marked as "not spam". Further correspondence from them, via the same address, ends up in spam list. Filter seems to be ignoring me reporting the misclassifications.

I can't tell whether the spam filter is personalized per user, or whether my reports are drops in the ocean. Either way, I get that feeling that I don't own my own email account either.

A while back, I realized I've missed lots and lots of important mails during about four months - Gmail had suddenly started flagging completely legit mail as spam, even from people I've had contact with previously. I tried flagging those mails appropriately, moved them to the inbox, but that didn't help either. Finally, I wrote a complaint feedback message, and a few days later, my spam filter was back to normal. I assume they put a small fraction of their users in A/B tests for the spam filter and can remove you manually on request. If that is true, it's completely unacceptable. I should probably migrate soon.

I think it's optimistic to think that they would have removed you manually. The test probably just ended or the bug was corrected.

FWIW I moved to FastMail from Gmail about three years ago and one of my main worries then was that I’d be inundated with spam when I didn’t have google’s filter to keep things clean. Since then I think I’ve only had one spam message come through to my inbox, and one or two that were incorrectly classified as spam. My worries were unfounded, as it turns out.

(No affiliation with fastmail, just a customer.)

Author here. Shockingly, the company is to blame as much as gmail in most cases.

There are a few settings to configure to be able to receive and send emails and many companies can't bother to do it. This can be as stupid as not setting the MX records or sending email from yorcompany.com (admire the subtle typo). Spam folder guaranteed.

In this article for instance, the issue is incredibly stupid and only surpassed by the triviality of the workaround. It's been known for years, yet nothing is done about it.

Why prevent users from whitelisting yorco.com? It's equally anoying that you can't blacklist what Google whitelists. You have to do folders or rules to delete mails from whitelisted senders.

I think because either yorco.com is a valid domain and fails SPF or it isn't a valid domain and allowing it lets anyone send mail from yorco.com. I don't understand why failing SPF doesn't reject the message in a way the sender can see.

Having the wrong domain, the forwarding SMTP server will fail both the reverse MX check and SPF. That's straight to spam folder, if not transparent deletion.

For gmail, you can register your domain in the google webmaster tools and it will show how many emails were filtered or not, by gmail users.

Why spam folder or delete rather than reject? IMO, such messages should always be rejected so that the sender can see that they aren't delivered. The usual explanation for spam folders or deletion is to prevent spammers from finding out what works, but this particular case that doesn't seem to apply since it is easy for spammers to check and not doing the basics should never work. Maybe in this case the point is to force you to use google webmaster tools...

You mean send back a rejection email? Where are you gonna send it? We just established that the sender domain was being impersonated or was not setup properly.

That's opened up to infinite loop of rejection emails.

No, I mean reject it as it is being sent (see SMTP protocol). As far as I know, all mail servers can do this except possibly qmail. qmail's out of band error messages used to be a big source of spam, although I haven't seen them much recently so I'm guessing it was finally patched not to do that. It didn't loop. From what Volundr said, maybe it uses DMARC now.

I know my ISP's mail server actually does reject at sending time since I am on the gcc mailing lists, which unfortunately pass along a fair amount of spam, and when it is rejected the list software then sends me the bounce message (rejected based on From header checking, I'm not sure of the full details).

Possibly Google does this too, I don't have direct knowledge, just going off what others said. But it seems like systems wouldn't stay misconfigured long if they were.

My main point is that IMO, mail that fails sender verification should not be delivered at all, not even to spam folders. Anything else is just making a bigger mess and helping spammers. Ideally the sender should be notified that it wasn't received. But I'm sure Google has good reasons for whatever they are actually doing and my understanding is that the main reason that sender verification has become mandatory as quickly as it has is due to Google pushing it.

If you have DMARC setup, it actually does reject it in a way the domain administrator can see. Google does send DMARC reports.

Thanks, good to know.

To be clear, none of the issues I was seeing were in any way caused by my incompetence. I had MX records, SPF records, DKIM signing, and still Google can arbitrarily decide to pass some of my mail through, and then refuse (or worse, hide) the fourth reply in an exchange with a customer.

I wanted to clarify that because the above comment makes the issue seem less important. It is important.

> I have frequent problems with their mail getting lost, classified as spam, rejected, filed somewhere, or just difficult to see.

I have the exact opposite problem. My clients with ISP email, free email, low quality shared hosting email have the problems. My clients with G Suite, and Office365 have no deliverability, spam or reliability problems.

I've had problems with Google putting their own messages to me in the spam folder.

The Gmail spam filtering is totally hit and miss - I have had a number of Gmail/Gsuite accounts and the level of deliverability is all over the place. When it's good, it's great. When it isn't, no action you take will prevent it from sending important messages to the spam folder.

Again, I just haven't seen this. Not just with myself, but with my clients. My latest email issue related to Comcast nuking form submissions from a client website ...randomly. I use "nuked" because unlike Gmail it didn't get delivered to a SPAM folder, it was just ... gone. Not all submissions, just a couple here and there, enough to drive the client crazy, and create mistrust.

Issue before that was related to Earthlink (yes that Earthlink) doing the same thing. my solution in that case was to put my G Suite email on the BCC list - guess who got 100% of the form submissions? > This guy

Gmail spam filtering is not hit or miss, it's very reliable both from my own experience and my clients, whom typically "upgrade" to solve spam and deliverability issues with other providers.

> Gmail spam filtering is not hit or miss

Yes, it is hit or miss, as our vastly different experiences show. Be happy you've gotten lucky. I have not. And I know I'm not alone as I regularly come across others who get confused when they have their first experience with this, because they used to have the same belief about Gmail as you do.

Note that deliverability problems are difficult to notice. Few people report or investigate, it is generally assumed that E-mail is unreliable, and that people often do not reply to E-mails.

The only reason I noticed is because I really cared about talking to customers and because I run my own server and can look in the logs to see if the E-mail was accepted by Google.

Setting aside the validity of the dollar amount, the title belies a dangerous view of product development.

In the real world, the buck stops with you. Even if it's not fair, if there's an issue your users are having, you need to fix it.

At their scale, there is no excuse for not polishing every minute aspect of the UX, and that it includes how it interacts with every email service.

Expedia stole 187m from themselves by not having their shit together.

We've encountered this a lot and when sending emails to organizations that use Google Suite or Gmail. When we explicitly don't want the messages to be grouped our team has began making the subject of the email unique to the request.

e.g instead of:

Subject:"Password Reset Notification" (or) Subject:"Website Support Request"

we'd use:

Subject:"Your Password Reset Request - March 14th 10:19am" (and) Subject:"Website Support Request - Jimmy Davis, Failed Login"

Yes, this is the exact fix the article suggests making. And then goes on to say that he never did it because it was too hard to format dates in Java or some such nonsense.

In fairness, it's a bad UX that forces you to change your own workflow to keep it from introducing undesired behavior.

It's bad UX to have messages that are entirely identical from a user perspective, but perform different functions. The email with the functionality to reset your password was no different than the past ones that have no effect.

The emails should have specified the expiration date in the text of the email document from day one.

>>> The emails should have specified the expiration date in the text of the email document from day one.

That doesn't avoid the issue. Similar emails are collapsed, it's not about being identical.

For example, two orders listing the (different) products you bought might get collapsed, despite being fairly different.

To be fair, the built-in java date libraries were notoriously difficult to work with until recently.

And don't forget about internationalization. There are more than one hundred languages to support!

I display the full password reset link in the email body which makes each email have unique content.

It still threads the email, but the last one will show up when clicking on it.

That might not actually be enough -- I've seen gmail collapse emails with distinct subject lines. The format was something like "[foo] Password reset" and "[bar] Password reset".

The "fix" for this is actually pretty trivial, and it doesn't require using a unique subject or sender.

When you send an email that you don't want to be collapsed into any previous thread on Gmail, include an `X-Entity-Ref-ID` header with a random value.

I don't remember where I learned this. I can't seem to find any official documentation mentioning it. But it works.

If the company can’t figure out how to add a date to the subject line, they won’t be able to figure this out either.

It’s gmails bug, but I’m amazed they couldn’t figure out hiw to implement the fix. Amazed but not too surprised given how not nimble big companies are.

It's not a bug.

To be fair, the header should be significantly easier to implement. There is a truckload of issues with formatting a date in 100 languages that don't apply.

Incredible how Hacker News figured out the workaround in 6 hours, when they couldn't in 6 years.

We've been using it for some time as well, not even with a random value, just `X-Entity-Ref-ID: null` works too. Found it through some obscure StackOverflow answer.

Or changing the sender's email address.

e.g. myaccount+randomstring@domain.com

Otherwise a unique subject is always easy for some people.

The issue is impacting all users who forgot their password, use Gmail (not sure about other clients) and don’t notice the hidden messages being at the bottom (it’s really hard to spot).

So the "bug" is collapsed message/sort/thread view?

Feels like this is about as un-bug-like as you can get. Adding the unique Subject: line feels like the right fix (in you) rather than shout at google "you have a bug"

The argument is that collapsed view should show the most recent message and collapse the older ones, not the other way round.

But it doesn't really know which individual messages inside you have read. So collapsing everything but the most recent one is dangerous because that is a real risk of missing messages.

The current way yeah you might have to see where the conversation left off- but it's the safest to not hide something from the user.

Gmail does know which messages in the thread you've already read, though; it'll leave the unread messages uncollapsed, but the ones that were already read will be collapsed.

The problem is that the first message is also uncollapsed, so the user has to know to scroll to the bottom.

I am quite surprised by this response, and even more that there are several saying the same thing.

> So the "bug" is collapsed message/sort/thread view?

No - the bug is that new messages were being hidden because they were mischaracterized as redundant.

I quickly disabled Google's 'conversations' feature (or whatever they call it), when it was first introduced, because it did not seem to be doing a good job, but did not think about its wider implications.

I probably should not be surprised about these reactions, as I have had experience with fellow developers who insist that what is manifestly a bug is not one because it is "working as designed", or because there is an undocumented workaround to the undocumented problem.

It does a great job, apart from this, it's a really annoying thing that I have no idea why they've done it.

While a whimsical writing and storytelling style sometimes works great, in this case it makes the article very confusing, even contradictory in places.

I also find their numbers very hard to believe. Expedia's net income last year was ~$400M. They could apparently make a 60-character fix and increase that by $187M but they don't, because...reasons?

I'm guessing $187M is referring to total bookings, which is in the $Billions

Where did it have you confused?

Changes can be impossibly difficult in large companies for no particular reason.

The defense of gmail in the other comments is bizarre. Gmail suffers from a number of similar, very serious usability issues. From the perspective of the user, failing to show the correct password reset email is a bug in gmail that affects many sites.

A less advanced non-threaded show-most-recent email view would not suffer from this issue. When you add a more advanced feature, make it the default, and inadvertently reduce usability, you are at fault.

I won't defend Gmail, but not testing critical email with the most popular client seems unwise. And, tracking failed resets for popular domains seems like a good idea.

Gmail's user experience is terrible in many ways, but I don't think it's fair to excoriate them for adding a feature that's not so idiot proof that not even one in a hundred users will misapprehend how to use it. That school of ultra-spacious user friendly designs for the lowest common denominator is how we ended up with the nightmare that is the current Gmail interface.

A few years ago, on Nu Gmail (Ajaxy Gmail) but pre-Inbox, I tried to walk my grandma through adding a contact.

Their add contact screen, with the form to fill in contact info, had two same-colored buttons, not that far apart, same copy. One added the contact whose info you'd just been putting in. The other helpfully erased it (add a new contact, was what that one meant).

That's probably been fixed, but the whole settings area was kinda that way back then—like someone slapped it together without once thinking about how it'd be used. IDK what it's like now now, I use basic HTML gmail.

[EDIT] who's to whose. I cannot believe I wrote that.

> The defense of gmail in the other comments is bizarre.

This statement is bizarre, I don't really see anyone defending Gmail here - just commenters offering further explanation on what's happening and other people mentioning specific examples of being caught out by this behaviour.

Maybe, but you shouldn't be getting any password reset emails anyway, because you shouldn't be forgetting your passwords.

What, you can't remember literally dozens of different passwords from different sites, that all have differing and incompatible password requirements?

password manager?

1) Doesn't work across devices, especially work computers. 2) Not really reliable. I'm constantly having to manually go into Firefox's saved passwords manager to look up a password because it just won't automatically put it in the form for some reason.

Even that doesn't eliminate the need to receive password reset mails. I've seen sites that suspect that their password DB has been compromised invalidate all existing passwords, requiring you to go through the password reset mechanism to pick a new password.

I had never used Gmail until a few month ago when I switched jobs and my current company uses GSuites.

The Gmail UI is horrible, the amount of confusion it creates and how illogical it does collapse and order things amazes me. Why not just show me the content as it is and let me figure out how to handle it?

You can disable conversations and view all individual messages in the chronological order without any grouping.

But it will not solve other UX issues, like the fact that gmail UI encourages top-posting, even if the sender replied inside the body of the original e-mail. (it skillfully hides the option to reply inline inside the quoted body, even in this case)

It also makes a complete mess when you write responses inside the quoted text, if you're not extremely careful and aware of this fact.

For example these are identical messages:

As gmail user sent it: https://megous.com/dl/tmp/gmail-garbage2.png

As I received it (text e-mail message gmail actually generated): https://megous.com/dl/tmp/gmail-garbage4.png

Now someone tell me this is not a complete garbage!

Honestly, conversing with gmail users is my least favorite thing, as long as they use the web client. Gmail webmail is not a serious e-mail client. It does not even implement threading.

Not OP, agree that the UI is still horrible in many ways, but still thank you!

Thanks for the pointer, will check changing the conversation view setting out. Yeah, I have noticed the issues with quotations too - it's really messy and hard to parse.

This title is way too clickbaity for what it's talking about: Gmail grouping emails by subject can bury password reset emails

Google should indeed fix every single UX issue, however minor, given its operation scale.

That said, such overblown criticisms reminds me of the quote "There are only two kinds of programming languages: those people always bitch about and those nobody uses". It's true for everything.

I just want my colorful threading back. Every time someone 'improves' the Gmail UI, it gets harder and harder to use.

Back when each new message in a thread had a different-colored header, it was vastly easier to use by skimming.

Now, with everything in various shades of monochrome, threads collapsed or expanded, quotes hidden or not hidden, signatures here and there, it's virtually impossible to tell at a glance where one message begin and the other one ends.

All Gmail would have to do to fix the issue is consider the link target when diffing to determine which part is quoted from previous messages.

I've noticed this myself on the occasion I end up requesting a password reset multiple times. It's annoying at best to have to click open a closed message and then click again to show the quoted text. I'm sure it took me a minute to figure out what was going on the first time I encountered this.

I opted for one email sent/received, display one line option in Gmail eons ago. The collapsed email format is complete bollocks with one small exception... coming back to work after a vacation. Collapsed view allows for reading a series of emails to acquaint with how a particular subject evolved over time while away.

So after vacation > turn collapse back on > clean up vacation built up email > turn collapse off again

Edit: oh, I also proactively search for "lost" emails with the following search string:

has:nouserlabels -label:inbox -label:drafts

With my workflow, I apply a label to everything I want to save when I move it out of the inbox, meaning I've finished dealing with whatever the email requires. So that search string finds whatever has fallen out of my workflow.

...okay, I have a very vague memory of this being straight up announced by Google as a new feature back around 2002-2005.

Basically, if I'm remembering right, there was a semi-standard header field in emails that used some sort of hash to identify which email was being replied to. This was how emails were threaded together in other clients, and worked well for the most part.

But there were some situations where it didn't work accurately, leaving some emails un-threaded, so Google created its new conversation view to both flatten the reply tree (so you'd see all emails so far before responding, instead of making the same reply as someone else) and group up emails sent from clients that didn't include the hash.

And now we're here.

Alternate title: Our Assumptions About MUA Presentation Turned Out To Be Wrong.

The concept of collapsed email threads is a terrible idea for the typical user. It probably only makes sense for public figures who get a ton of unsolicited emails (whoever designed Gmail must have been a public figure). It also would explain why Marissa Mayer redesigned Yahoo mail to have collapsed email threads by default.

The average person doesn't get so many emails that they need to have them collapsed and sorted based on the sender. Most people don't have a problem reading every email in their inbox each day - In fact, that's what they want to do.

"Debugging complete. It’s not a bug, it’s a feature.

"The direct impact of this bug"

"Actually, the bug is still active"

So I thought the author realized it was a feature, not a bug. But then the author goes on to claim it's a bug, several times. It's clearly not, it may be a shitty feature but it's not a bug.

It took me one instance of this to realize what was going on. Why only one time? Because I took the time to learn how my tools work.

Someone else's mailbox isn't your tool.

Additionally, the search function doesn't easily surface the most recent messages matching a query. "Gooz Frabba" seems to get me a short list (the first ever mention from years ago; the one where I mentioned it to a friend; etc), followed by a random assortment of messages which don't contain the very latest email I received until I scroll down a screen or few.

Is the GMail behavior described here accurate? Does GMail really hide the _latest_ email in a collection sharing similar sender and subject?

Actually, GMail expands the most recent email and collapses older messages. (I just double checked.) So the premise of the article seems a bit misguided.

I know it hides similar parts of a series of emails because it cuts off my email salutation and sometimes people think that I'm being disrespectful. I wish I could stop it from doing that.

If true, I wonder how similar these emails were to have that happen.

GMail will hide parts of a message that are similar to previous ones. When the entire message consists of what GMail considers to be quoted text, it doesn't really look like there's an email there. I just tried this myself and in the second password reset email, I only see a button with three little dots that says "show trimmed content" when I mouse over it. It's tiny and I wouldn't expect someone to figure it out if they didn't spend some time looking or have an idea of what happened.

OMG, it's not just me. It's super frustrating on mobile. On certain threads, I'll literally spend 5 minutes just trying to find the new content.

I still miss pine, tbh

(If this reference is too old: https://en.wikipedia.org/wiki/Pine_(email_client) )

You can use the basic html version of gmail: https://mail.google.com/mail/u/0/h/1pq68r75kzvdr/?v%3Dlui

I use this - it's way more responsive and faster than the current gmail UI.

My only gripe with the basic html version is that the back button is broken when you're trying to go back to search results after clicking on an email (you need to click on "Go back to search results").

> I still miss pine, tbh

So use it then. alpine (the successor to pine) is my daily email client. I currently have 161,210 messages in my inbox, it handles it just fine.

It's still supported, and has new features like UTF-8 support and viewing HTML emails (without the images).

It's better than any other email program I've used with one exception: No images in HTML email (although you can view an HTML email in your browser, the attachments don't come along).

offlineimap and mutt work just fine for me...

It might work well for you, but

  are you sure your recipients
  aren't
  seeing emails that look like
  this?

Sending plain text emails that look good when nearly every email client is mostly only tested with html emails is actually very difficult, and sometimes simply impossible. Format=Flowed doesn't always fix this. Plain text emails are defacto deprecated just because a lot very popular email clients don't handle them well.

One major problem with plain text emails is the standard was created when nearly everyone had a screen wide enough to display at least 78 characters per line (or whatever it was). Today people spend most of their time on screens much narrower than that. FF is a half assed solution to that problem

html was designed to work on a much wider variety of screen sizes so it works much better for email.

I say this as an offlineimap/mu4e user who knows some of my recipients are seeing bad formatting and doesn't really care.

> I say this as an offlineimap/mu4e user who knows some of my recipients are seeing bad formatting and doesn't really care.

This basically describes me (though substitute "bower" for "mu4e").

[edit]

It occurs to me that it would be relatively simple to write a filter that converts text e-mails to a multipart with an html e-mail having the same text.

Still no solution to replying to html e-mails inline, but the intersection of (Doesn't topost) and (uses html e-mail preferrentially to text e-mail) is relatively small.

Author here. The one you got is yet another bug. It's also possible that the behavior changed since.

In the case I described, the email is not opened with only 3 dots showing, it is instead completely collapsed. Like if you clicked the header to collapse it manually.

I can’t believe I got hooked in by that deceptive, clickbaity headline. Am I on the ExpressOnline and didn’t realise?!

I've always felt that gmail is more cumbersome and packed esp in mobile! Inbox was nice in few areas however the core gmail app on mobile is almost impossible to get to a correct mail.

Enter Outlook mobile, absolutely love it, no wonder its the most rated mail app in app stores!

This happened to me and was incredibly frustrating. Figured I was just a moron, but perhaps not.

tl;dr - Gmail collapses 'threads' of identical messages and you might not see the newest one, and on password reset links (and others) that can be a problem.

At Blekko we ran into a variant of this caused all of our regular logging reports to get thrown into a single gmail thread and collapsed (making it hard to find the results) so we changed our logging script to include the date in the subject line.

The fix for the author would have been to change the subject to 'change password request received on 14-mar-2019' or something similar (you can include the time too for more uniqueness). Then it always starts a new thread and the messages are always visible.

If only there was a way for email messages to indicate they were part of a chain or not…

Sarcasm aside, it stinks to have to workaround email clients that deliberately don't pay attention to convention. Putting the date in the subject should be a clear indicator that Gmail is in the wrong here—there's already a date field in the headers.

I have experience this exact issue described in the article so many times. It is so frustrating when trying to reset passwords because other than the time stamp, each email looks identical.

I have had too with some support emails. With Gmail it’s really easy to not see some messages in long threads. Also reading a thread is quite difficult with gmail hiding parts all the time.

It's odd to me that the phrase, "conversation view" is nowhere in the OP and it's literally the name of the feature they're talking about, which can be turned off.

That doesn't matter from the perspective of the sender. Most people won't turn it off.

I switched off collapsing Gmail threads, or having them as conversations I believe Google call it. It's quite difficult as I would find myself struggling to find certain emails.

it doesn’t matter if gmail is buggy or whatnot. you have a duty to your clients and shareholders to fix the issue. in most cases this means circumventing an uncontrollable entity. in expedia’s case it means having pseudo-random email formats when requesting a password reset. as a long time so-called “hacker”, this is junior-level stuff.

very interesting read - for an alternative interface to email, try mutt[1]

[1] https://smalldata.tech/blog/2016/09/10/gmail-with-mutt

Is this not just a repost of the flagged post from yesterday?

I want to know! Did 9,000 people actually book the listing?

some friends thought I was insane when I said i preferred outlook to gmail purely for UX reasons. The email threads are so difficult to track.

Outlook (the client not the service) has a feature that could cause similar issues - the focused and “other” tabs. I’ve missed a few responses to AWS support requests because of it.

On the other hand, I chose to use the Outlook client instead of the built in mail client for iOS, specifically because there is a grey area between “spam” and automated email that isn’t urgent but I still want to see.

Most of the time Outlook gets it right.

Very interesting read. Author should have submitted the problem and solution as part of a bug bounty perhaps. Something for us all to watch out for.

Put the time of the request in the subject line of the emails so they are unique and don't get threaded.

Wow, I just saved you $100M! /s

TLDR: Gmail hid the most recent password reset email with a working token link at the bottom due to grouping similar looking emails together. The fix was to introduce a date in the subject line, thus preventing grouping in the first place.

Edit: Well, actually it didn't work and it is still an ongoing bug.

The "still not implemented" fix, apparently.

"Actually, the bug is still active. I never had the opportunity to fix it, didn’t stay long."

Reply on news.ycombinator.com