Google backslides on federated instant messaging, on purpose?
fsf.org
fsf.org
The sad thing is that I no longer consider this unthinkable. There's an extremely disconcerting email monoculture emerging around gmail. Practically everyone I exchange email with uses gmail. Companies and universities are switching to gmail. It can be difficult to get your mail accepted by gmail if you run your own servers: I help administer the servers at one organization whose mail (personal correspondence, not mailing list posts or the like) often gets sent straight to gmail's spam folder and we're doing everything right in terms of DNS/SPF/DKIM/etc (in fact, the same config works great elsewhere). Try to look for help on gmail's website and all you can find for this problem are the "Bulk Sender Guidelines" - as if the only people who aren't using gmail already are bulk senders.
Now consider Google's actions as of late. I could totally see them one day saying, "we're not going to accept email from you unless we've emailed you first or you've contacted us to ask for permission." There would be outrage, but I also wonder how many people would actually stop using gmail if they did this.
Edit: I'm not saying this will happen, I'm just saying it's not unthinkable, which is sad.
Google apps is fine if you are a small company and just want to get some email working but I've always found it hits limits pretty quick.
As a counterpoint, I've been running my own mail server for 15 years (from various hosted and dedicated servers) and I haven't ever had a problem delivering to gmail...
Yahoo! is already doing something like this http://help.yahoo.com/l/us/yahoo/mail/postmaster/bulkv2.html
How likely is it that the same company which builds bleeding-edge machine-learning systems to track and predict our behavior online, and which uses these AI predictions constantly to maximize their ad revenue, somehow cannot find a better way to filter out spam invites?
How likely is it that the same company that houses the likes of Hinton, Norvig and Kurzweil under the same roof can't find a better way?
Google is packed with experts at solving the "spam filtering" (i.e., pattern recognition) problem.
It appears this was done on purpose[1], driven by a corporate culture that no longer cares as much about openness. [Please read jholman's responses below. He's right, I went too far with this last sentence.]
--
[1] http://mail.jabber.org/pipermail/operators/2013-February/001...
--
Edits: added "it appears" at the end, to tone down the language. Also, reworded and added sentences to make my point clearer, and corrected text to refer to invites, not messages (thanks for pointing that out, mdc!) and point out that this was indeed done on purpose.
It does suck that they did this. I have been hit with many spam requests through Google Talk recently though. You should at least be able to whitelist people in your address book or something. Yeah, you can still send them an invite, but what if the third party's service adopted the same policy as Google?
They should at least let you opt in to requests.
Only Google manages to constantly produce falls positives (including mail from, sweet irony, Google services like Analytics) and regularly allow spam and phishing mails through.
The other two, ran by relatively small providers, are nearly perfect.
Don't overestimate Google.
(The same "intelligent" Google also seems to be unable to figure out which language I use, despite me telling them on a regular basis.)
Add to this ‘temporary addresses’[0] and train your spam filter on everything send to invalid such addresses and basically nothing gets through.
[0] I use a scheme where my website and mailing list addresses are of the form claudius_YYMM@example.com. Mails to these addresses are marked as spam after the 15th of MM+1 and before the 15th of MM-1. Obviously only works for mailing lists if they’re open to non-subscribers as well.
They both get several times more spam than the much more recent business-only Google address, yet if their filtering lets through one per month it's a lot. I can't even remember the last false positive.
The majority of the mail that ends up in my Google spambox consists of legitimate email from reputable sources (Amazon, Facebook, Google itself), and barely any actually spam. Extra annoying: perfectly fine email from our own services regularly gets flagged as spam by Google, and we often have no f-ing clue why.
And don't get me started on Google Groups spam filter, which for some reason is even worse. I have to turn it off for any group-address I want to make accessible to non-members.
As an anecdotal example, I have an email address that's been strewn about the internet for almost 2 decades. It's currently hosted on Google Apps, but it has a non-Google domain. I get a spam message in my inbox maybe once every couple of months.
I also have a gmail address. I never use the thing. But it gets inundated with spam, and every month or so when I go look at it I have to clean lots of junk out of the inbox.
Since they're both hosted by Google, I'm forced to conclude that the gmail one gets many orders of magnitude more spam merely by virtue of ending in @gmail.com.
Out of those 100k connections, one or two emails come through to my valid email account. So yes, scattershot makes sense, but from looking at my logs, unless your account includes a lot of numbers you aren't going to get hit :P
And why do people keep making excuses for Google?
Google simply isn't very good at filtering spam, something most regular ISP's can handle perfectly well, and the lack of options in Gmail and Groups clearly show that they don't care very much about it either.
This allows the ESP to curtail spam problems on their end (for example, Mailchimp heavily throttles your emails or outright bans you if your spam rate creeps past a very low percentage). It's an all-around good thing for the ecosystem, with the exception perhaps of publishers that get the unlucky "spam instead of unsubscribe" user action.
But Gmail does not participate in this loop. They don't tell any ESP that a user has marked an email as spam...that data all stays in house. Why? Hell if I know - perhaps they don't want to tip off spammers to being detected. On the flipside, reputable ESPs get less leverage on spammers in their network.
You could perhaps increase the requirements for sending invites, such as having the recipients server send a CAPTCHA, although spammers seem to be able to get around CAPTCHAs anyway. Perhaps there would be some other solutions that I haven't thought of.
Per said, a month ago, "is there anything you can do about it in that case, otherwise we will have to institute very tight limits of invites per day being sent from federated domains", speaking about specific domains, and speculating about a possible future strategy.
FSF says (paraphrasing): "we have this symptom, we're convinced it's for this technical cause... " (so far so good) " ... "and this email thread says that Google is doing it on purpose". Bullshit, that thread says nothing of the kind. Stick to what you know.
My point is not that this is okay [0]. My point is that FSF is claiming "Google is doing this on purpose and this link says so", and YOU said "this link says so", and that's not what that links says. So stop making shit up. Stick to what you know, and/or to your opinions.
FSF could have avoided my complaint by changing "According to this thread, Google is doing this on purpose" to "Based on this thread we're guessing Google may be doing this on purpose". You could have avoided my complaint by leaving your whole last sentence off.
[0] Supposing that this IS what's happening, I'm not defending it (nor opposing it, I'm not forming an opinion). But as a side note, my understanding is that nearly all email providers DO do this thing for email; if a given domain is spamming hard enough, eventually email providers start dropping mail silently. Don't they?
However, the problems with this solution are a) that it doesn't discriminate between domains with a lot of traffic and domains with a lot of spammy traffic (though, again, that's just speculation and maybe they do have that data), and b) there's no message to the user that is being ignored. I don't know if jabber supports rejection messages, but it would be much better if they could get a message that let them know why they were being turned down so they could pressure their chat server operator to reign in the spam accounts.
It appears from the FSF post that they've actually tested invite requests, though, so presumably they are rejecting all invites, which makes the earlier email even less relevant as evidence of what is going on now. Surely the FSF has a contact at Google?
It's really unclear what's going on, actually.
We don't know exactly what the FSF is seeing, nor if they did fair tests, nor if they're reporting fairly (though I tend to assume so, tentatively).
We don't know if Google did something deliberately, nor who did it. We don't know how it relates to Per's mailling list message, if at all. If there was action, we don't know if it's an experiment, or a bug, or who it affects, or how often.
More generally, it doesn't seem like this is an urgent issue that requires panicking. The world can get along fine for a month without people being able to invite gmail users. But, on the other hand, if the worst case is true, that Google has started blocking invites from ALL other domains, ALL the time, then I think the FSF's political position is sound.
So far, though, this really isn't like Reader.
Spam detection algorithms, even Google's, are rarely perfect. They probably came to conclusion that blocking foreign invites is a good tradeoff. We don't have enough information to evaluate if it was a good tradeoff. I think it was. I was recently getting lot of spammy chat invites (chat bots that tried to convince me to do an online payment for something), it was quite annoying.
So long as it's temporary (and given you can request whitelisting in the meantime), I don't really see the problem.
Google users have apparently been flooded with subscription requests from spammers, and the flooding suddenly became massive. The problem is, there are a large number of jabber servers out there which have open account registration without captchas. Most jabber server software doesn't come with a captcha module included by default, and of course, most admins don't bother changing defaults, even while running a public server with open registration.
Unlike some other comments here, I don't think Google has any malicious intent in this. This seems like a stop-gap measure, while they figure out and implement a proper solution.
As to the proper solution, the XMPP community is largely moving towards having captchas, or other forms of verification, and there are a number of proposed standards.
The thing to understand here is that the XMPP community has historically not had a spam problem. Due to the nature of the protocol, spoofing wasn't possible from the start, and there were no large lists of JIDs for spammers to abuse, so things worked out fine for a decade despite a lack of captchas. The good news is that the XSF was already preemptively working on the spam problem, and the speed with which XMPP specs (XEPs) get defined, implemented and deployed in servers and clients is far faster than any other large scale open protocol that I'm aware of.
However, I've been getting a consistent barrage of requests from spam email accounts to chat; accounts with obscene names like 'sweety+69+for+free@freemail.ru' or something to the sort. The names/domains of each are different every time, so I can't simply block a domains.
I ask: if Google does not block outside requests, what could they do to stop this sort of thing?
Over the years, Microsoft has tried various attempts at making MSN/WLM interoperable with AIM, XMPP, and ICQ -- albeit with limited success.
The argument is a "the technology isn't there yet and spammers are running rampant, so the obvious solution is block it, which isn't ideal", not "Google is doing this because they want a monopoly on chat, and broke the protocol to do it".
Doing this strikes me as throwing the baby out with the bathwater.
But before they start on this research project, they might want to do something about the spam their users are getting now, right?
What is it with modern times, where people are willing to sacrifice fundamental things, like applications that adhere to standard protocols, to gain some minute level of relief from something that's just "annoying"?
They're not stopping anyone running a personal ejabberd server.
One of the first things that one thinks of when designing/projecting a large-scale system should be the potential for abuse. Are you trying to convince us that Google didn't take that into account when they chose XMPP as the underlying technology for GTalk?
Imagine spam filtering based only on the purported sender address. It'd probably suck.
Have a separate "spam gchat invite" folder that you could review periodically to look for false positives.
I can't see any downsides and it definitely solves the spam problem. If they could make it apply to other gmail accounts as well, even better. I've gotten annoying gchat requests from sweety69@gmail.com as well.
This isn't perfect being the enemy of the good, it's a description of why it's not even good.
It's been like this for years.
Edit: that came out a little curt. The OTR protocol is different than what Gmail calls OTR. Gmail's just turns off logging; the protocol is encryption.
I've met two very intelligent women (That don't know each other) that for most things have very reasonable opinions.
But throw "child abuse" and it is like telling a robot to hide in a corner in a round room, they just break and if they could they would pass laws about instantly killing suspects of child abuse.
When I try to argue with them of how "x" or "y" is bad idea because of its side effects, they always reply: "I don't care, EVERYTHING is worth doing to protect children."
And then I understand why so much politicians use "for the children" rhetoric when they want something.
They redirect their frustration by wanting to punish "child abusers" when the source of this fictional abuse is in their head.
I guess that's what happen. The room is round, but their upbringing tells them to stand in a corner.
I prefer "well intentioned but misguided", "worked up by a sensationalist media that obscures the fact that kids are safer than in the 'idyllic' 1950s", etc. theories.
At least in this case. It's a textbook case, exactly like the one in South Park Uncut.
There's lots of other possibilities.
Sometimes there are different classes of users with conflicting interests, and no algorithm or ML model or abundance of good intent can resolve them.
Google doesn't have to perfectly solve the spam identification problem in order to make IM usable without rejecting all remote messages.
Why won't this work:
- Allow IMs if the sender's address is anywhere in the recipient's contact list, or in a separate IM-specific whitelist (see next point).
- Have a separate IM whitelist that supports user@domain or @domain for special circumstances.
- Automatically add any foreign outgoing-IM recipient either as a new contact in the sender's contacts (if an IM field is added) or in the separate IM whitelist, so return IM from the same address is guaranteed.
- Have a list of unconfirmed senders from the last N hours/days (after which they expire and are dropped), with a web-based interface... (clarification) notifications of these requests would NOT push through to the Google user's IM interface.
- Optionally reply to unknown senders: "You are not a confirmed sender of this recipient. Contact recip OOB to ensure future delivery."
- Optionally add an IM field to google contacts, for instances where a remote address doesn't function as an email address.
I always ignore them..but it's a very annoying problem. You can simply ask for user confirmation, you're bothering me about something I really don't want to deal with.
In real history, this is like telling kids that Nintendo used to make Japanese playing cards.
Not really, Nintendo cards weren't used by most countries on earth minus China and didn't revolutionized gaming world widely the way google did with organizing knowledge.
Well, they also made love hotels and a (taxi) cab business. Without making any predictions of my own, I agree it'll be pretty interesting to see what a company like Google will become down the road.
Sure, Google could spend a lot of time trying to come up with a technological solution that doesn't break federation (except for in spam cases), but it would be difficult to do on a service that fundamentally doesn't make money.
This sounds like blog trackback all over again - useful, a nice idea, but nearly worthless once spammers figure out how to pee in the communal pool.
No, it’s not. It’s this kind of constant exaggerated claims that give a bad reputation to people that speak for free software, and make them look like out‐of‐touch conspiracy theorists (think Stallman).
Now, I’m not saying I disagree with the message as a whole; it is bad that google closes the door to this kind of interaction, but it’s nowhere close to what they claim (and in bold). With their example, you’d effectively only be able to speak to the person on the other side, but never receive their messages. This case is very different, as only the initial contact is unilateral (the person with the gmail account has to invite the other one), but after that, it works just as well.
It’s this kind of stupid exaggerated argument that drives people away from your message.
Also, not having OTR messaging (ie. the socialist millionaire protocol, not Google's private mode) just seems stupid in this day and age. And so does not using TLS for federated messages, if what I've just read in another comment is true. It really is time to switch.
I'm assuming you're on an android device (you didn't say) -- all¹ Android devices are "always" connected to GTalk already. GCM (C2DM) sends all push notifications/wakeup requests over this channel, and therefore the framework ensures the socket remains open even when sleeping.
¹ - ones with Google Services
http://mail.jabber.org/pipermail/operators/2013-March/001610...
For example, Facebook allows accessing their Chat servers using XMPP, but they don't support federation at all.
I'm sick of this kind of language, why does EVERYONE who does advocacy have to speak like this? Can no one be reasonable?
Again: people, run your own shit. Stop playing the google roulette.
No, really. I would rather spend that five hours making sure shit works with my boyfriend, or my family, or my friends, or learning something that will make me money, or working on a pretty cool hack I thought of while in the museum last week, or working on my unfinished novel that has been languishing, or work on proofreading a paper for a friend, or just relax.
Yeah, system administration is under those priorities.
I am more than willing to pay. I just want to pay one person for everything. I want that integration. I want webmail. I want to be able to access it from my phone without issue. I like integration.
I may be a geek, but I have better things to do than figure out why my email isn't getting to a friend because someone's IP address ended up on spamhaus.
I understand it's not so easy to run your own stuff, but it's also not as hard as some people believe, especially if all your need is a small setup for personal use.
I'll try to wrap up some tutorials in the near future to help people run stuff on their own. I'll remember to ping you when done.
That's not to mention that I'm seeing 1GB RAM minimum to run the thing. That's a lot of wasted resources for one person.
I can run my own email server. I just have better things to do with my time.
Whatever you do you must pay, with time, money, your privacy or control; I see you've made your choice.
> I am more than willing to pay. I just want to pay one person for everything. I want that integration. I want webmail. I want to be able to access it from my phone without issue.
I'm in the same boat - currently looking for somewhere to move stuff to. I'm giving atmail.com serious consideration for mail/calendar/contacts, as the first is reasonably easy and the latter two _seem_ to be working on desktop, android, and ipad. They don't do XMPP though, so I need to figure out a solution for that.
I would still recommend Zimbra or Zarafa, as far as open source goes. Or if you can't be bothered with running your own, both products list a series of partners who do this for you.
You might argue that a few hours a month isn't much time for working email, but it is when Google will reduce it to 0 for you. In a hierarchy of needs, food is far more important than email, and despite this very few of us try to completely manage our own farm.
Getting an email server up and running is trivial, just like planting a few crops in the garden is. It's the reliability and maintenance that is the hard part.
Except if you're running your own chat server you won't be able to connect to Gtalk users.
1) Discontinuing Google reader
2) Retiring CalDAV support
3) Removing ad blocking apps from the Play Store
4) Replacing Android chief Andy Rubin with Sundar Pichai.
5) Dropping support for sending chat invites to Google chat users from other domains.
Wonder what else is up for the next couple of days.
It's interesting though, so much of google's core business relies on trust, and they've been making a lot of moves lately to just throw away as much trust as possible. Maybe it's sustainable, but I'm skeptical.
A bunch of people disagree with the FSF though.
* Mozilla can switch to a different default search engine (like Bing) and still get as much revenue without depending on Google.
* Google can't act like it controls Mozilla, because they don't want to lose the partnership either
Now that Yahoo is powered by Bing, there are only really two big multinational commercial search engines. Both of them are owned by companies that also offer browsers to compete with Firefox. If neither of them paid, Firefox would still have to include a default search engine. I hope Mozilla can keep going, but honestly it does seem rather dependent on Google's good will.
I've heard this a lot. Are there any specific anecdotes that really show a strong analogy between the two? Do I have to assume that today's "Cloud" is yesteryear's PC-Compatible?
Google's social endeavors have always shown them to be out of touch and incapable of engaging the market with a product that people care about (orkut, knol, buzz, wave, and now plus). Their approach to "customer service" for many of their products is to actively disconnect from their users and let the natural release cycle take care of things. They have a long history of ugly interfaces with poor usability. Youtube is a good example, discoverability of new content has only gotten worse over the years and many fundamental features are really very broken or at best poorly implemented (such as playlists), despite being an otherwise mature product. They've killed off lots of small products and projects for the sin of not fitting in to the grand strategy. And they've tried as much as they dare to force people into using google+ whether they want to or not.
Google still has a huge number of world-class engineers, and that will enable them to continue making some truly great things over the next years and perhaps decades. But fundamentally google is on a track toward becoming just another run-of-the-mill mega-corp, and if they continue along that route eventually a lot of the talent is going to evaporate away (as has happened and is continuing to happen at MS) and there won't be anything to save them from mediocrity.
I think the answer to that can be found by watching Futurama episode 64, "The Why of Fry".
Looks like anything "free" is quickly disappearing : )
et tu, Google?