Websockets 101 | Hacker News Reader

Websockets 101

lucumr.pocoo.org

289 pointsthe_mitsuhiko50 comments

Encryption is not a panacea. As more websites use HTTPS, more corporate networks are installing content filters that passively filter HTTPS content.

Yes. I just said corporate networks are spying on HTTPS connections.

It's really simple to implement. Get some kind of web proxy that has support (Websense is pretty popular, amazingly) and generate a root cert on the box. Then use a group policy on your AD server to distribute the root cert to all the client systems. Now Websense can decode the HTTPS traffic by issuing its own fake certs to clients and handle the "real" HTTPS handshake on the frontend proxy side.

Result? Your websockets are still going to get fucked with by proxies. Can we please stop building fake protocols on top of real protocols now?

Interesting, I actually hadn't considered the possibility of an employer issuing fake certificates to peek at its employees' encrypted sessions.

For anyone unnerved by this, you can still get around it by routing your traffic via SOCKS proxy to an external host, but of course this assumes that 1) your employer isn't blocking SSH, 2) you have the authority to install PuTTY and Firefox (not sure if Chrome has SOCKS proxy support) and 3) you have an external server to proxy through (if you have a Linux machine somewhere, you're probably already configured to allow this). And make sure you route your DNS through SOCKS as well.

If they're proxying HTTP traffic usually they'll block SSH and other protocols. To work around this use an HTTPS/SSH multiplexer[1][2][3] or a simple reverse proxy to accept HTTP[S] connections and connect them to an SSH server on the backend. For the ssh client use a ProxyCommand tunneling app[4][5][6] to turn an HTTP[S] proxy request into a two-way socket, and you have an SSH connection over HTTP proxies.

[1] http://www.rutschle.net/tech/sslh.shtml [2] http://www.pond-weed.com/multiplex/ [3] https://github.com/stealth/sshttp [4] http://zwitterion.org/software/ssh-https-tunnel/ssh-https-tu... [5] http://proxytunnel.sourceforge.net/ [6] http://www.agroman.net/corkscrew/

This won't work with our games. We verify that the certificate was created by our CA and reject anything else. Someone would have to patch the game to bypass that. It also won't work on mobile phones that people use themselves.

How? Is this some JavaScript magic i'm unaware of? Because the actual connection from the proxy to your site uses your site's valid certificates; only the client to the proxy uses a "self-signed" one. Are you using client certificates?

We're more in the area of things that actually are not websites than websites. The websocket stuff I added as an experiment and if people want to use our stuff to make HTML5 applications without a proxy server in between. If it's indeed between browser and website someone could MITM it of course if that person can also give you a fake security certificate into your browser.

But on the large scale that will never happen. Maybe in corporate networks but I doubt that this will become widespread.

But on the large scale that will never happen. Maybe in corporate networks but I doubt that this will become widespread.

Maybe not ubiquitous, but multiple governments already use fake certificates to spy on secure connections, and it is incredibly common in corporate environments.

I would be surprised if this passed muster in US courts, that an IT dept could present fraudulent information to an individual in order to access private data, vs just blocking that traffic.

From what I've heard, US employees have very little legal right to privacy while at work, even if doing things or visiting e-mail accounts of a personal nature. Regardless of whether the practice would survive a legal battle, it is already widespread.

Unless I'm mistaken, a cursory examination of the certificate supplied to your browser should show that it is one of those "mitm" root certs and not the certificate for the website you are browsing. That should make it simple to at least know your encrypted traffic is being hijacked.

A cursory examination of the certificate will appear to be exactly the same as the "real" one, with the exception of lengthy obscure keys and numbers that nobody would identify unless they compared them to the original side by side.

The proxy vendor MAY provide its own issuer description in lieu of faking the original, but it by no means has to. Fake certs can look just as authentic as real ones if your browser trusts the root cert that signed them.

If you want to verify a certificate is signed by a "real" trusted CA, use a program that traces the chain of trust against a list of CAs that ship with browsers. I have such a tool here: https://github.com/psypete/public-bin/blob/public-bin/src/ne...

Couldn't you compare the fingerprints of a cert as accessed from a suspect network and from a trusted network?

Well yes, if you've got the same server and cert. Often the SSL frontend of big websites serve multiple certs so you'd have to have all of them to compare against.

Plugins like Convergence will verify who signed a cert for you, assuming the notaries aren't blocked by your corp proxy (and currently blocking any of them will kill the whole chain of trust, so it's not as useful as it seems).

In any case, your HTTP client having the right software and right root certs is the only way to know what is valid and what isn't. You could compare the fingerprint from your 3G phone's browser (or the issuers in the chain's fingerprints) to the cert from your PC's browser. Unless the phone is managed by your company too, in which case they can install the fake root cert there as well...

What's also kind of funny is it's easier to spoof a cert on a mobile phone than a PC. Phones can be updated over-the-air by the mobile provider while your PC has to give some kind of admin rights to your ISP. Since mobile devices are "the future of computing" this makes who controls the mobile device a scary proposition, especially in countries with oppressive regimes (or countries that like to shove internet legislation down your throat without asking).

I noticed a company I worked at had installed their own root cert on the machine they gave me. I disabled it, but didn't get any warnings about invalid certificates, so I assume they weren't using it to snoop HTTPS traffic.

Unless there's a way to detect whether a cert is installed somehow?

What you are proposing is generally illegal in the U.S. Companies are allowed to snoop on non-encrypted web traffic, because there is no expectation of privacy, but different standards and rules apply to encrypted sites, i.e., including banks.

Also, since bank websites and email are the most frequently accessed encrypted sites, a company would be exposing itself to ruinous liability if a data breach led to the release of private employee information, i.e., bank records. No company would take on that sort of liability, and you can bet that any competent legal counsel would make the company aware of this before they implemented something like this.

Very nice readable article. Having dabbled with implementation a bit I agree with most of his points.

The author complains about websockets being frame-based instead of stream-based, because he argues if he would want a frame-based system he could easily model it on top of it.

I think frames are awesome. It is easier to build a stream on top of frames than it is to build frames on top of a stream in my opinion (you don't need an extra protocol for starters). Besides that almost anything you would like to do can be expressed in frames, saving most of us a lot of headaches.

I would like to add that websockets are not the end-all answer for web-game programmers. It makes some stuff easier, and reduces some overhead, but it's still on TCP, meaning realtime multiplayer games are still as good as impossible to implement.

Sadly all the issues with proxies go for UDP and then some, so I don't think we'll see any ubiquitous browser UDP protocol anytime soon :(

> I think frames are awesome. It is easier to build a stream on top of frames than it is to build frames on top of a stream in my opinion (you don't need an extra protocol for starters). Besides that almost anything you would like to do can be expressed in frames, saving most of us a lot of headaches.

Maybe I'm just old fashioned but I really don't like the idea of frames forced onto me. I much rather get my data in small pieces and can already start processing data as it comes in than having something buffer up everything and then giving it me as a large chunk. Streaming json for instance over websockets is a lot harder than TCP directly.

IMO the built in framing is one of the things I really like about websockets. I'm sure it will become my transport of choice for non performance critical messaging even when a browser is not one of the peers.

My issue with it is that TCP is already a stream based protocol on top of a frame based protocol.

This is a frame based protocol on a stream based protocol on a frame based protocol. Now if you want a stream based protocol you would need to add another layer on top of that. Madness.

WebRTC is already there if you want UDP.

well does it though ? Data channels seems to use SCTP over DTLS over UDP which would create a sudo "TCP" layer on top of the UDP layer as I understood it :(.

SCTP? That's awesome, I happened to have written an SCTP implementation for my BSc final project and it's a great protocol that makes in-order-delivery optional so it's a great replacement for UDP.

edit: I hadn't heard of WebRTC at all, it looks like a dream come true, not only does it offer SCTP, it offers P2P in the browser, which I believe is going to enable Web 3.0 (yes I'll go there). Sadly they haven't even begun on implementing the Data Channels yet, which is the SCTP part of the spec.

I hope ordered messages will be optional. For gaming and analytics out of order udp is a good enough tradeoff for the better "latency" of not doing acks. The p2p stuff is awesome.

If anyone finds themselves needing to write a Websockets implementation, there's an awesome protocol test suite at http://autobahn.ws/testsuite

I wrote a C++ implementation for a side project and with the aforementioned test suite I actually found it pretty easy to get to 100% compliance. There's some ugliness in the protocol because of proxies, but it's definitely not the worst protocol in the world. The only big missing feature is compression and there's a proposal for that (you could certainly do application level compression, but I'd rather avoid writing compression code in JS).

Browser vendors must start to move away from top-down innovation, where they hoard APIs such as TCP and UDP and release sub-standard specs in their stead. Instead they must expose only the bare minimum OS low-level APIs (TCP, UDP, POSIX), keeping the surface area as small and powerful and direct as possible, and then let open source grow around this. Innovation needs to be decentralized, bottom-up. Not designed by committee.

For that to happen, we need to stop conflating "web apps" (trusted, installed, annointed with machine power) with "web pages" (accessed by single link click). At present, Web Apps are suffering and crippled by being lumped under the same security policy as web pages. But Web Apps need to have access to raw machine resources in the same way that Native Apps have access.

Those that don't seem to care for any of this insistence, tend also to be naive as to the massive differences between TCP and WebSockets, and IP and TCP and the whole stack in general. The WebSocket spec is a good example of people doing things in the most indirect way possible, with a maximum of red tape, as opposed to people doing things in the most direct way possible, with a minimum of red tape.

The Web as we have it in these respects is very much Animal Farm and 1984. There appears to be little thought leadership from the major stakeholders in this regard. People like Tim Berners-Lee are asking for change (http://lists.w3.org/Archives/Public/public-webapps/2012JanMa...), but the new incumbents don't seem to want to see.

I don't understand the point you're trying to make. Are you against standardization? What specific actions would you take to "minimize red tape"? You do know that TCP, UDP, and POSIX were all developed by committees right? How exactly is the Web in any way like Animal Farm or 1984?

Standardisation for the web targets too much surface area. Browser vendors need to agree on a smaller set of more powerful, more dangerous, more direct APIs so as to put the responsibility for innovation back in the hands of developers. Developers are being wedged further and further away from bare metal, with no right to return, even if they should so wish - this process needs to be reversed.

Raw TCP was considered for WebSockets, however briefly. Due to security reasons, WebSocket was designed so that it's impossible for a WebSocket client to spoof any existing protocol.

I would argue that things are actually swinging to the opposite direction right now. WebGL is very close to OpenGL ES 2.0, Web Audio API includes low level features, WebRTC 1.0 is slated to include datagram channels... Recent standards efforts are striking pretty good balance of flexibility, interoperability and security, IMHO.

I am referring to networking and storage, not graphics or audio.

At first glance it looks like the top-down massive surface area standards approach is making progress. But the APIs are really sub-par when compared to TCP, UDP, POSIX and what open source could do if given the chance to grow around them.

The security reasons keeping raw TCP out of the browser apply to web pages, not web apps (trusted, installed, annointed with raw machine power to act on the user's behalf). This means that it is not possible to build a SMTP/POP/IMAP etc. client in the browser without the use of a third-party proxy (which introduces additional security concerns). This is a terrible blow to web apps. WebSockets are a diversion.

WebRTC is a herculean effort. UDP should be beneath WebRTC not bundled alongside it. Again, web page security issues have been projected onto web apps, hence no directly exposed UDP.

There is no decent offline storage mechanism in the browser. No POSIX. No fsync. No way to build any kind of performant database. IndexedDB is the only thing available. It is poorly designed to begin with and implementations at present are slow, buggy or lacking. It looks good in the to-do list demo's but beyond that is a pain to work with. I have seen Chrome's IndexedDB reboot Windows over and over again on occasion. All the great open source database implementations are locked out. Everyone is forced to grow over IndexedDB. Much better if LevelDB were directly exposed. But POSIX is what is needed.

The major issue going forward with regard to web apps, is that they need to be seen as distinct from web pages, and when installed by the user, given access to the last 40 years of computing progress.

Excellent article, thank you for sharing.

As someone who has implemented a WS server and client and would like to be able to host them in Amazon's cloud, the notes about ELB are worrysome.

Websockets through ELB works fine if you use the TCP listener. You lose out on some HTTP-specific features like Cookie-based session stickiness, but it works great otherwise. We've been using Websockets/ELB in production for almost a year now.

Why do standards like WebSockets don't include anything that can be used in pure HTML? This would be a great tool for building efficient dynamic apps, if you could "submit" forms without going to a new page, and then get incremental page updates as a response.

I'm not sure I see the use-case there as being more useful than JS based WebSockets. Writing a general JS library to do what you suggest would be pretty straightforward, and the server-side work wouldn't be any different (serving partial HTML based on request type).

If your users are worried about security / performance enough to turn off JavaScript, they probably wouldn't want "HTMLSockets" enabled, either.

If your users are browsing with a browser so old it doesn't have a reasonable JavaScript engine, then it wouldn't support "HTMLSockets," either.

Basically, there's nothing you could do with "HTMLSockets" that you can't do with WebSockets, and with only a small layer to make it so. However, there's a whole host of utility that you get from WebSockets that you could never get from HTMLSockets, or would require a huge amount of hacking in order to get it to work.

And all this is beside the "pureness" argument, where one might say that HTML is for data, CSS is for design, JavaScript is for behavior; WebSockets are behavior, not data.

This is like asking why we should bother having any (new?) HTML or CSS standards at all, when you could write a custom rendering engine in JavaScript for every website.

There is tremendous value in having a solid, usable foundation for web development that doesn't rely on an imperative language to work. To start with, declarative features are semantic and allow for semantic upgrades on wide-scale basis. Do you like your Firefox/Chrome spell-checking in text areas? Do you like them being resizable? Well, this wouldn't be possible if all textareas were some JavaScript hack. (Hey, with only small layer you could actually fake a textarea. Does that seem like a good idea in retrospect, though?)

Also, it would be great to be able to develop simple dynamic web applications (or prototypes) without doing something complex. Progressive enhancement is buried far too early.

If your users are worried about security / performance enough to turn off JavaScript, they probably wouldn't want "HTMLSockets" enabled, either.

I frequently browse with JavaScript blocked by default, and I can tell you right here that this assumption is invalid. There is a world of difference between JavaScript library and a standard, declarative technology. The latter is more likely to be faster, have less bugs, no side-effects and be secure.

Because it's a sockets layer. You could very easily send HTML over it if you want.

What is so bad about a little Javascript?

And, how does one get sufficiently customisable incremental updates without having to standardise (and implement) a huge complex new language to control them?

What is so bad about a little Javascript?

What's so bad about having a declarative way of doing something that a lot of people will find useful?

And, how does one get sufficiently customisable incremental updates without having to standardise (and implement) a huge complex new language to control them?

By using element IDs. Every element in the incoming frame that has and ID that matches and ID of an existing element replaces it in the document. People have been doing this for years. It's called AHAH (Asychronous HTML and HTTP). Doing this without any custom code and with every update being as cheap as sending WebSockets frame would be absolutely awesome. It could be used to create very sophisticated and dynamic applications with very little complexity either on the client or on the server side.

This exists in Firefox. HTTP Server Push. https://en.wikipedia.org/wiki/Push_technology#HTTP_server_pu...

An even quicker introduction to the final specification of WebSockets: http://socket.io.

P.S. I'm aware that a) this doesn't help anyone not using Node.js on their server (it's not even part of my production stack at work (yet!), even though I'm bringing it to light here) and b) it's more than just WebSockets, for instance it will gracefully degrade on legacy browsers.

I just can't miss an opportunity to sing its praises because it has so many benefits over the simple implementation.

It is not actually limited to Node. https://github.com/learnboost/socket.io/wiki

Though some of these libs do tend to lag in version support.

(Shameless plug...)

I have a list of websocket libraries and a small echo server demo here: http://ajf.me/websocket/#libs

That first link there to your Github repo is broken: https://github.com/TazeTSchnitzel/WebSocket.us

Oops. I moved it to my main website's repo: https://github.com/TazeTSchnitzel/ajf.me/tree/master/htdocs/...

For folks thinking about using Socket.IO for websockets, it works very well, but remember that most open-source implementations ONLY implement WebSockets and NONE of the other fall-backs that are in the Socket.IO spec. So without some work, these libraries are often useless for mobile development, for instance.

I'm using SockJS (client and server: https://github.com/sockjs/sockjs-client) for project I'm working on, which does implement a variety of fallback methods (iframe tricks, JSON polling, etc.) and it's been working pretty well (some work better than others.)

For those who to opt for these libs, I recommend the following for protocols/fallbacks:

protocols_whitelist: ['websocket', 'xhr-streaming', 'iframe-eventsource', 'iframe-htmlfile', 'xhr-polling', 'iframe-xhr-polling']

This gets you a working system in pretty much every modern browser supporting WS, and /usually/ works pretty well in IE8/9. Just try to avoid page refreshes in those two browsers (very spotty results if you don't - I'd recommend putting up a message for your users) and implement the workarounds for finicky browsers (things like catching ESC in Firefox.) I've been pretty happy with it so far.

Man, I would have loved to have this about a year ago. Although then I would have missed out on some of the hilarious drama on the websocket mailing list (e.g. http://www.ietf.org/mail-archive/web/hybi/current/msg08668.h...)

Pop-up Video Excerpt:

The author (Armin Ronacher) also authored Flask (Python web micro framework) and its template engine Jinja2. I've used Flask for a number of my most recent projects and really have enjoyed working with it.

Back to your scheduled broadcast!

Loving the article! Just thought I'd point out: "needles complexity"

> Messages sent from the client to the server are obfuscated with a basic transmission mask of 4 bytes which is sent within each websocket package.

What is this? Some pretend security feature?

The purpose of the masking is to prevent malicious applications from being able to send arbitrary bytes over the wire, for fear that they may be able to screw with buggy proxies.

Reply on news.ycombinator.com