Hacker News Reader

Comment by pbhjpbhj | Hacker News Reader

Perhaps this is the point we need legislation to require that IM services with above X users in the relevant jurisdiction must provide a full, open, protocol and enable interoperation via the published protocol? That's the only way I see the companies involved would move to enable cross-platform communications?

Aside:

>It's a wonder we can send e-mails between many servers.

I have a shared account on a small ISP, Microsoft won't receive our emails, even when replying to an email from a Microsoft user, even when user whitelisted. [It's probably the same with some other suppliers?] The only way to reply to people using Outlook/Hotmail/Live (I'm a user of hotmail for something like 16 years) is now to send via MS servers - basically had to make an account for this purpose.

The sending IPs aren't blacklisted with Senderscore (or other blacklists, I mention senderscore because they apparently provide domain reputation services for MS); some "related IPs" have medium reputation scores (senderscore 72).

View on HN

I run my own e-mail server too and constantly run into issues:

http://penguindreams.org/blog/how-google-and-microsoft-made-...

If I send a new person an e-mail, I have to contact them on Facebook and tell them to check their spam folder. Fuck Google and Microsoft and their terrible e-mail filtering. Just because people are so dumb they will open any ransomware sent to them doesn't mean you drop all e-mail you don't trust with no way to account for it.

How long have you been sending mail from your latest IP Address?

I run some websites that have their own mail servers. One thing you didn't mention which can help is to be sure to use an email signature. Basically short emails are more likely to be flagged as spam, especially if there's an attachment.

In use 12 years, only ever used to reply (except when messaging businesses), never done mailings. Of course there is a footer with bricks & mortar address, phone, unsubscribe link.

Had problems with hotmail before, it's just associated IPs being factored in too strongly. Same ISP has an IP address that spam has been sent from; support responds with won't fix. In their favour at least they responded.

Domain has SPF but not DKIM.

Thankfully no other mailbox provider blocks us. We could pay to be added to the third-party "friendly" domains list of course ...

I'm surprised they wouldn't whitelist our domain but leave it on a tight leash.

Why don't you properly configure DKIM and SPF? Then you won't have this problem.

From the fine article he linked: SPF, DKIM and DMARC are all domain verification systems for validating e-mail’s origin to prevent spam. I have all three records set in DNS records for all the domains I send e-mail from, verified they were correct using testing tools, and I still get flagged as spam.

Google uses a lot of signals. The article above is on http, I saw a drop in spam flagging when the web host on my email domain was https only.

I'm not a big fan of legislating technical standards, but in this case I think I could support it. IM has become a fundamental communication mechanism. Imagine how held back society would be if we had 50 different voice standards rather than the singular phone system we have now. IM needs to be forcefully standardized.

Korea mandated through legislation that all online shopping had to go through ActiveX controls, because that was the only way to guarantee secure encryption at the time. Now it's practically the least secure way to online shop and even Microsoft recommends not using ActiveX, but it's a huge amount of work to strip that infrastructure out and make everything work 'properly' so it's still an issue even today.

In short, legislating a solution is a workaround that can turn out to be a huge detriment not too long down the road. I'm no libertarian, but it seems like a bad idea to me.

That's a good point - especially important if someone's thinking about legislating a particular technology. On the other hand, maybe in this case it would be enough to legislate a requirement for openness of the protocol? I.e. whatever it is, if it has enough many users, it has to be fully available to third party integrations.

What about SMS? Everyone has it, it works, and is usually fairly instant. It's not very feature rich, but could there be an SMS 2.0 protocol that was backward compatible?

Isn't this RCS?

This feels reminiscent of net neutrality and also general competition laws. I wonder if there's something in that which could be drawn upon?

Sounds like a straw man, just because South Korea made a mistake doesn't mean legislating a standard is bad.

That was interesting to google.

The first link I got: https://www.theregister.co.uk/2015/04/02/south_korea_to_depo...

Except we still do have many different standards for the phone system. As an example the main codecs used for encoding voice today are μ-law (North America), A-law (Europe), GSM (2G/3G), AMR (3G/4G), iLBC (VoIP), G.729 (VoIP).

That is just for voice, now imagine you are a phone company and want to send a call to another phone company. There must be so many different things you need to cooperate on to make it happen - billing, switching, physical connections, numbering, codecs, etc. Yes the actual protocols may be standardised, but the main part is cooperation.

It's not standards we need, it's companies that are willing to cooperate and work together. None of the big tech companies want to do that now though. Everyone wants their own walled garden so they can say they have more users that X competitor.

(I'd argue that XMPP is good enough, or with a few extensions could be made good enough, for a company who really wants to push open IM)

It's not something needed. It's a plus, even if it influences society a lot, it's not something like food or healthcare, where if someone plays unfair people die. And it's not like net neutrality either, because you are not costrained by space (putting cables underground). People should be free to choose a service, and if the server want to stop providing the service or change the way it does ut, it should be free to do so. Anything else you can use contracts, or change provider, as long as there is net neutrality in place. And it's not something that messes with your health, so no justification for regulation aside from an arbitrary and unprovable "society being held back". Being unable to make something because of regulations, that's some real and provable holding back of innovation.

Don't we already have GSM and LTE, and who knows what else? And I'm not aware of any legislation for the use of those over others. As far as I know, as long as I have the right to use the frequencies, I could start my own cellular telco that transmits whatever format I want.

GSM and LTE use public goods (spectrum).

Spectrum is owned by the public, and regulated as spectrum is a limited resource. IP messaging isn't by any practical means a limited resource.

> As far as I know, as long as I have the right to use the frequencies, I could start my own cellular telco that transmits whatever format I want.

No. Spectrum is regulated in terms of what technology you can use, and kind of usage is supported. You can't buy LTE spectrum and broadcast radio on top of it, and vice versa. You can't (in most cases) take 2G spectrum and deploy LTE on top of it.

> No. Spectrum is regulated in terms of what technology you can use, and kind of usage is supported.

Interesting! I was not aware of that. Thanks!

Also, bandwidth is a function of how much spectrum is available (for example between 1900MHz and 2000MHz there is a chunk of 100MHz) and the spectral efficiency (for example, 1 to 2 bits per Hz available).

For example:

If a carrier "owns" the block between 1900MHz and 1940MHz, and let's say LTE supports 2 bits/Hz using Frequency Division Duplexing (each direction, up and down, gets a chunk of spectrum), with 3 sectors per cellular tower, the carrier can support a total of 40x2x3 = 240Mbps per tower, or 120Mbps each way.

If the tower serves 1000 users (they serve far more in dense urban areas), each user will have 120kbps of capacity, and if they all use it at the same time, that's the speed they'll get. If there's a single user, the maximum down speed for that user would be 120/3 = 40Mbps.

> As far as I know, as long as I have the right to use the frequencies, I could start my own cellular telco that transmits whatever format I want.

Point I ("I have the right to use the frequencies") directly depends on point II ("whatever format I want"). When applying for using a certain frequency band, you have to specify usage and purpose, and you can't just change your mind, i.e. you can't apply for GSM frequencies and say you will be running a GSM network, but then run the PavelLishinOverAir protocol.

>When applying for using a certain frequency band, you have to specify usage and purpose, and you can't just change your mind, i.e. you can't apply for GSM frequencies and say you will be running a GSM network, but then run the PavelLishinOverAir protocol.

While this is mostly true, in practice (at least on the 450-470 MHz band I work with), you can give fairly vague purposes, like "telemetry" or "SCADA", and it doesn't matter what vendor or what protocol you use. In fact, there is no section of the licence that says "I am going to use protocol X on equipment Y manufactured by vendor Z" or any part thereof (emissions designators are a different story).

I don't have any experience will cellular licences, though. Given that carriers frequently repurpose spectrum as technology changes, I can't imagine their licences say "GSM only" or anything like that. The band that used to be 3G GSM is being used for 4G LTE on my phone right now.

This likely differs a lot between countries (though there should be a common core due to ITU of what is being regulated how by whom); in my country the cellular frequencies were auctioned off for dozens of billions of euros to the cellular service providers, with strings attached, e.g. that the networks had to at least use and support 3G.

The same regulatory body (BNetzA) is also responsible for licensing telecommunications providers in general. This is probably the crunchpoint - if you're not a licensed telecommunication provider, it's simply illegal to offer telecommunication services.

Huh, I didn't know that. Guess it's time to pivot PavelLishinOverAir to become some sort of drone taco delivery service.

Right, the point is that you don't have the right to use whatever frequencies you want because the usage of the frequency spectrum is highly regulated, and for good reasons.

Basically none of which apply to IM.

> I'm not a big fan of legislating technical standards, but in this case I think I could support it

I feel another such example is IPv6. It could move much faster if it would have been mandatory already.

China has effectively mandated IPv6, and IPv6 certainly is popular in China (I think all mobile phones in PRC use IPv6, I may be wrong) - but let's not forget that China is promoting IPv6 because APNIC's allocated IPv4 space for China was comically undersized.

Unfortunately this is not the fact. While some ISPs are experimenting with IPv6, the vast majority are not on that train. Indeed, IPv4 address space has practically run out in China already, but many ISPs are taking the nasty NAT-like approaches, Carrier-grade NAT [1] for example. And it is particularly common in mobile. So far, IPv6 is only (relatively) well accepted among universities (CERNET [2]).

[1] https://en.wikipedia.org/wiki/Carrier-grade_NAT [2] https://en.wikipedia.org/wiki/CERNET

The Great Firewall likely affects the reliability of everybody's stats, but China seems to be way behind the US in terms of IPv6 deployment. (<3% versus 30%):

- https://www.google.com/intl/en/ipv6/statistics.html#tab=per-...

- https://stats.labs.apnic.net/ipv6/CN vs https://stats.labs.apnic.net/ipv6/US

Nope. If mobile network uses ipv6 then people would have no problem accessing Facebook

You do realize how firewalls work, right? You have iptables and ip6tables. Sure you need entries for both, and sometimes ISPs forget, but it's easily implementable in all major routers and operating systems.

Firewall rules are not yet implementated on ipv6 network in China by ISP. It is a fact, and actually people are using ipv6 network as a means of bypassing the wall.

And everything else would have slowed down because of resources (human and other) devoted to ipv6, and the complexity of ipv6.

I don't think it needs legislation as such but could be worked from a licensing point of view.

Everyone moans about AGPL. Fine. Make something better that says if you're going to interoperate with a service, you better damn well actually interoperate. Make the license viral like GPL and hopefully when we all push in that direction, everyone needs to join up.

> IM needs to be forcefully standardized.

This is totally crazy to me. It would be like the government legislating when and where people have spoken conversations with each other.

No, it's not. What you're proposing would be like the government saying you can only use this specific service, which may be run by the government, to communicate with others. Standardizing IM is making is so that if you want to talk to your friend across the country you're able to using different applications but the same protocol. Forcefully standardized would be mandating any messaging app that's used in the country would have to follow the protocol. It's almost the same thing that happened to fax machines in the 70s and 80s before the Japanese government stepped in.

How would this look different to SMS, except that the protocol might be more feature rich.

Though I don't think the legislature even uses IM let alone understand it and understand it technically.

Keep it simple. All protocols for communication and data formats for storage must be published as open source or public domain. This is to prevent or reduce lockin, walled gardens, etc. No other change. They can still compete in numerous other ways.

Although this is an appealing idea, I don't think it is realistic. Lobbying would destroy any legislation almost immediately, and there would still be the issue of competing standards.

I think a more achievable version would be that any online service which maintains user data must provide a method for a user to retrieve that data in an open-source format. Some companies would deliberately obfuscate formats, such as calendar appointments as JSON rather than iCAL or another existing standard, but I think it would give users more control than they currently have.

"Lobbying would destroy any legislation almost immediately, and there would still be the issue of competing standards."

Oh, I agree. You're describing the effect of a corrupt government allowed to prevent or eliminate laws meant to protect its voters. I was talking about what law would be necessary if voters cared enough to push one.

And the result is OOXML, which is well known for being a trainwreck and which nobody - including Microsoft - has been able to implement properly, despite being an "open standard".

"And the result is OOXML"

The result of open-sourcing all data formats and protocols would obviously be all the data formats and protocols released as open-source. OOXML is not the only one in existence.

My point is that companies forced to do this would create abstruse formats, and then even misimplement them, to thwart those trying to interoperate.

Which would be obvious, flout the law, and allow for legal action to be taken.

The company itself has to use the format for live message exchange, have to publish a full working protocol document.

With OOXML Microsoft aren't required to be able to read & write a files correctly when it's written to the protocol. As I envisage it the requirement would be to open the protocol actually used "on the wire"; again with Word Microsoft don't use OOXML as primary.

I think it's workable except perhaps in requiring political will. It's an internet age equivalent of allowing third-party car parts manufacture.

Will it be obvious? How? Browsers often fail to implement open standards like CSS, even when they have nothing to gain, it's just that shit's hard.

Microsoft don't use OOXML as primary.

Sure they do, that's what .docx files are. They've been the default since Office 2007.

Microsoft was compelled to create and implement the standard so that governments can comply with their requirements to use "open standards". And they followed all the steps, including getting it past ISO, despite the mess that the format is. And so governments can now tick the "open formats" box when purchasing Office. It's all a sham, and I don't see how it'd be otherwise with this proposed law.

Oh yeah. They'll probably try that. That will either be defeated with tons of sweat as it largely was in products like OpenOffice or the legal recommendation modified to attempt to deal with it. It might also be handled in the courts if the law includes language that specifically instructs to attempt simple, readable protocols to "facilitate interoperability." Obfuscations could be argued as the opposite of that in court.

That will either be defeated with tons of sweat as it largely was in products like OpenOffice

Then why do you need open standards? Every major proprietary IM protocol has been re-implemented by open source software, including Hangouts: https://bitbucket.org/EionRobb/purple-hangouts

It might also be handled in the courts if the law includes language that specifically instructs to attempt simple, readable protocols to "facilitate interoperability.

XMPP, the thing Google is being roasted for dropping support for, is often decried as being very complex and hard to implement correctly. If genuinely open protocols often end up like this, how can you possibly punish companies for writing complex, hard to implement protocols?

"Then why do you need open standards?"

"This is to prevent or reduce lockin, walled gardens, etc."

What I said. I'll add that it's hard to copyright API's to sue over compatibility when the API's and their implementation are released as open-source software.

"If genuinely open protocols often end up like this"

What are you talking about? There's all kinds of results from protocol design. You chose one of the worst ones. I was a PSYC fan over that if I had to choose an IRC replacement or open chat standard. A test for new ones might be trying to run it through LANGSEC's Nail, Cap n Proto or ZeroMQ to see if they handle it easily. Anyway, it's easier to do compatible implementations if you have source and can't be sued/imprisoned for imitating it.

World you be happy to put the offenders in prison? Or if they didn't pay fines, to show up and collect?

i kinda doubt anyone sees protocol violation as a criminal offense.

it's wild to me how quickly "we should find a market-external pressure source for forcing interoperability given the profit motive seemingly guarantees lack of federation because it relies on competition as its motive force" becomes "we must frogmarch innovators to the death pits"

The correct way to solve this problem is through raising awareness of privacy issues and supporting efforts to create competitive and attractive open source communication protocols, not legislation enforced by a prison/military industrial complex. What you're suggesting is to hand the State further control over our choice in communications platforms and trust them to never abuse it.

I have been helping out a charity that uses a small web hosting company for email.

Any emails they send to Gmail automatically go into the spam folder.

Why? I reckon it is because there is no SPF configured, no DKIM configured, etc.

We could try to get that small web hosting company to configure all that.

Instead, the plan is to move to G Suite for Nonprofits.

And what if they don't publish the protocol? Do we block downloads of their client?

Even if we assume they are American companies, and we can sue them if they violate the rule; how on EARTH would we define an IM protocol broadly enough to avoid an easy avoidance of the law, but narrowly enough to not basically block all proprietary protocols?

Good questions.

If a company doesn't publish the protocol in use within, let's say 3 months of a valid request by a third-party then they'd be liable to a fine, perhaps something like $1 per day per registered user, plus costs of government and registered third parties in pursuing the interoperability request.

Small companies might escape, but if Facebook tried it they can't really run away. You'd need to be able to issue notices to ISPs to block those convicted of breaking the law, such orders are already part of UK law (against torrent discovery websites for example). Again you'd get some leakage but big providers like Microsoft couldn't feasibly expect enough customers to move to VPNs to bypass such a block.

IM client definition could be hard but in practice you'd have clauses requiring a judge to decide if the traffic exchange amounted to an IM communication. I see no problem with having a broad definition, if the definition were narrow then it would just get worked around. The definition would probably include email as a subset.

Were there other person-to-person communications protocols you are concerned would fall in the scope that definitely shouldn't be open?

> You'd need to be able to issue notices to ISPs to block those convicted of breaking the law, such orders are already part of UK law (against torrent discovery websites for example)

And most of us in the US hope we never lose the right to open access in the way citizens of the UK have. I can't think of a worse thing to happen to the internet. The UK has proven that once you allow a government to censor the content of the internet they will aggressively and rapidly expand that power to censor and control.

I can think of a few... one example that comes to mind is a video game. The communication protocol to synchronize the game could fall in scope, and I think video game companies have the right to have their own proprietary game network protocol.

I have a personal mailserver box in a cheap colo, and gmail won't put messages from me to my gmail-using friends in their inbox. It's shameful what email has become.

Yeah, that's one of those problems which are hard to solve without global collaboration, and current major IM participants are too selfish and greedy to understand that concept.

Historically giving the government more power to legislate the structure and content of digital systems has been disastrous.

Are you implementing spf?

Reply on news.ycombinator.com