Hacker News Reader

Comment by IceMetalPunk | Hacker News Reader

Let's be clear about this: there may be a good reason for that. Most spam filters use some sort of Bayesian analysis to determine what's spam. Whenever someone marks an email as spam, it updates its probability mapping based on the email's content.

In other words, if gmail is filtering more conservative emails as spam, it's because gmail users are marking more conservative emails as spam and teaching it to do that. In other words, the bias is that gmail users don't want conservative emails more than they don't want liberal emails.

Which, means, hey, if you're a conservative who doesn't want your email filtered out as spam in gmail, maybe consider saying things in your message that aren't annoying to gmail users instead of blaming The Algorithm

View on HN

Except that the study found that yahoo and outlook leaned the other way, except not as much. Gmail was uniquely left-leaning.

Which I addressed: it's based on the user base. There's some sampling bias to consider here, in that the demographics of people who use Yahoo, who use Outlook, and who use Gmail are not equivalent.

Your explanation seems purely speculative; do you have any reason to believe that it's actually true?

Experience, perhaps?

That maps partially to my own experience. The majority of folks I know who identify as 'conservative' are older, less tech savvy, and tend to use whatever they had set up a long time ago. @AOL? @Yahoo? @MSN? If it's working, they'll just keep using it.

A few I know have made a transition to gmail, I think at the behest of younger children (as in, they may have set it up for them during a holiday visit, etc). But in my own life, with the majority of conservatives I know where I also know their primary email, it's not gmail.

My experience agrees with you and I just realized that you can roughly guess someone's age by their email domain. As a Millenial, I've used gmail for going on 20 years, but I feel like that was more or less the peak of email - I don't know that Gen Z uses anything else. My guess would be they moved onto social media and while they probably have emails, it was never their primary method of online communication.

I would wager that for most Gen Z-ers (and younger), their emails are used only for two things: confirming account registrations and communicating with older people (a la teachers, businesses, etc.) Most of their inter-demographic non-realtime communication is likely social media and texts.

https://www.statista.com/chart/9937/the-gmail-yahoo-mail-age...

https://www.pewresearch.org/fact-tank/2017/03/20/a-wider-par...

I would imagine the demographics are different just purely based on the fact that Yahoo Mail has been around for longer than Gmail.

I mean, different providers have different features that attract different types of people. But if you want quantified data, there's at least this that I found with a quick Google, showing a significant difference in age demographics between providers:

https://www.statista.com/statistics/547531/e-mail-provider-r...

And there's quantified evidence that older people tend to be more conservative:

https://www.chicagobooth.edu/review/there-are-two-americas-a...

So that alone is worth taking into account.

It might be true, but how likely is it that users are more important than the spam detection algorithms? I don't think this user data is sufficient to exclude/disqualify other explanations.

"...how likely is it that users are more important than the spam detection algorithms?"

I'm not sure you understand how spam filters work. No one is sitting there hard-coding all the possible terms and phrases that indicate spam; that would be unfeasible. The spam filters learn what content is considered spam based on user activity and user flagging. It's not that users are "more important than the algorithms," it's that the algorithms are trained by the users in the first place.

In more general terms, the algorithms represent user preferences, so the user demographics directly determine the algorithms' behavior.

>"In more general terms, the algorithms represent user preferences, so the user demographics directly determine the algorithms' behavior. "

It all really depends on the algorithm used. I am familiar with many algorithms which could be used in these cases, and I'm not sure whether Google is using a 'stock' algorithm, or as I suspect, something tailored to their requirements. Each algorithm will produce different results, so which one you choose has a huge impact on what is 'filtered'.

The user-base has a substantial impact, but you could probably get a variety of results by applying different algorithms to the same user data. You implicitly asserted that user demographics were the dominant factor, and I am not sure that's true.

Different algorithms do indeed learn from data differently. But the entire purpose of a spam filter is to filter out spam, a word which is defined subjectively. Any spam filter which acts contrary to its users' collective preferences would be considered bugged and broken, regardless of how it decided that, because it would mean it's not doing what it was designed to do, and not learning correctly.

Gmail spam quality is internally evaluated based on the ratio of moves into and out of the spam label. So you're completely correct: it is optimized for how its users collectively define spam. There is no objective definition of spam.

The spam detection algorithms are trained ON the user's reports, so it's certain.

Which algorithm you choose, what data is used for training, and how the data are used are all very important. The user-base has an impact, but I am not sure whether it is the primary cause of the discrepancy between e-mail providers.

Another very scientific study on the subject: https://theoatmeal.com/comics/email_address

> https://www.statista.com/statistics/547531/e-mail-provider-r...

Why does this study exclude ages 35 to 55? Seems like a big gap to me. So this study is only comparing Zoomers, Millenials, and Boomers, and completely excludes Gen X.

I was curious about this, I have 2 conservative emails in my spam folder, one sent from a colo facility, and one sent from amazon ses. Both seem to have adequate dmarc and spf records, so I can only imagine that it'd be something in Google's filters.

Personal anecdote: I've given my details to both Republican and Democrat parties here in Texas. The only email list I ended up on the Democrat side was a local congresswoman's mailing list (but I've been phone-banked a ton). On the Republican side, I've wound up on 2020maga email lists, something called "middleamericanews", Texas GOP, something called "americasfreedomfighters", etc. A bunch are really low quality e-stores. I can only surmise that (at least here in Texas), there are differences in how the two parties partner with their mailing lists.

I imagine IP geolocation probably plays a role in where these groups send emails in the first place. I'd guess someone with an email account most accessed from Texas would be sent far more conservative emails than, say, someone with an account accessed from California or Manhattan. But this is definitely just my own speculation.

When I contact my senators and representative about literally anything and give them my email for a reply, they end up spamming me constantly with their newsletters bragging about how they're keeping Obama, and now Biden, from being able to get anything done, or how they would like to make the lives of immigrants and young women harder.

Howdy, Spam folder. GMail learns a bit more about how much I don't want to hear about the above.

(My senators are Cornyn and Cruz, representative is Carter.)

Other example: innocuous-looking, apolitical site with a few nice knitting patterns, and the promise of more if you register an account. I used a custom email address just for them in a domain I own.

Two weeks later: Newsmax and other far-right stuff to said email address that had never been used for anything else. No, I'm not worried about men calling themselves women and assaulting my unsuspecting daughter in a bathroom.

More education of the Spam filter.

Sometimes I'm annoyed at the GDPR, especially when implementing it. But this thing just won't happen in Europe anymore, at least from my experience. Especially when writing to your local representative it's unbelievable that they are allowed to start messaging you regularly with their advertisements.

New laws against stuff like this in the US generally have exceptions for politicians, so it wouldn't help.

Which I'm sure has nothing to do with the fact that politicians are the ones making the laws............................................

(I wasn't sure how many ellipses would convey my exhaustion accurately, so I used them all.)

>> "they would like to make the lives of immigrants and young women harder"

Attacking us queerfolk seems to have come back into fashion, too. And, bizarrely, furries. I wish the "Beto is a furry" campaign had been a fluke. Wishing it would go away is the one thing furries across the political spectrum agree on. At this rate, Nancy Pelosi will buy a knockoff fursuit off AliExpress in solidarity.

Oh, yes. Sorry for forgetting about y'all - I translated "traditional family values" a bit too narrowly and self-focused. (Paid parental leave is apparently an attack on the "traditional family".)

"(Paid parental leave is apparently an attack on the 'traditional family'.)"

Well, duh. Everyone knows that being barefoot, pregnant, and in the kitchen is a woman's natural place, so she should stay home taking care of the kids with no pay while her husband brings home the bacon to buy her pretty things, so he doesn't need parental leave either. Traditions must be right and unchallenged, otherwise why would they be traditional all those years? Obviously.

(This is the second or third time today I feel the need to write this disclaimer, but: everything above is, hopefully clearly, sarcastic as fuck. Please read into the irony the exact opposite of what the words say.)

Ideally, shouldn't this be customized to the user, so that liberal users get conservative messages sent to spam and conservative users get liberal messages sent to spam?

The problem with that is an individual user doesn't generally mark enough things as spam to give the filter enough data to perform well on. It's the aggregation of every user's spam-marking that provides a large enough data set to make good predictions. If they switched to individual spam filters, a lot more spam would make its way into users' inboxes.

Not necessarily. After a bit of customer-specific training data, users could be grouped into cohorts.

So to give an extreme example: there would be one cohort of unsuspecting elderly people that can't recognize spam and therefore still get a lot of Spam?

Yea,that would be a weakness of the approach for sure.

So, train a classifier to create an embedding of Spam Types, then train a different Bayesian filter for each Spam Type with only the data from the users in that type class?

I mean, yeah, I guess that would be possible, but is it a good idea? Echo chambers tend not to be productive, so if the choice is "get rid of stuff that most people think is bad" or "get rid of stuff that most people who agree with you think is bad" -- and if it takes much more work to do the latter -- I can't justify putting in that work. It's counter-productive, and all to satisfy peoples' desires to hear what they already think.

Ideally users should get what they sign up to receive and nothing more.

It's way too expensive to have individual models for these things.

The implicit assumption here is that 100% of the training data comes from regular Gmail users and it didn't start off with any seed data provided by Google employees.

Not at all. It doesn't have to be 100% of the data, it just has to be enough of the data that it influences the filter's decisions toward one direction over another. And since GMail has been in existence for 18 years and has a massive user base, I would be extraordinarily shocked if Google's initial seed training set isn't totally overshadowed by user-supplied data by now.

It seems fairly useless if we get different results depending on where we start from. That's not what I call learning.

To be fair, there would be a difference for awhile, as more training data makes its way into the system. But you're right that an effective machine learning system, or even just an effective Bayesian analysis, would be problematic if it didn't trend towards a the same answer as more data is obtained. And after 18 years and millions of users, yeah, if the initial training data still held any influence over its decisions, it would mean the whole system is broken.

Reply on news.ycombinator.com