Tunnel TCP Through WebSockets (CLI Tool) | Hacker News Reader

Tunnel TCP Through WebSockets (CLI Tool)

github.com

101 pointsderhuerst71 comments

First, as always hats off to the authors for publishing code and hacking on stuff for us all to use.

That said...You could just use corkscrew.

https://wiki.archlinux.org/index.php/HTTP_tunneling

Corkscrew + SSH has been helping individuals escape restrictive corporate networks for years.

Pretty sure SSH for remote server on port 443 plus corkscrew offers more than this solution. (Socks proxy, etc. Very flexible. No need for DNS muckery you would need with this solutions example etc.)

Web browser JS can't use HTTP tunneling, unfortunately.

And it's all about the web browser, so...

Is it? At least my client is for the command line.

socks proxy

Thanks for the tip! I stumbled upon corkscrew when quickly researching possible solutions, but eventually, I didn't have a second look at it.

I use since a year https://ondevice.io which does ssh and others via websockets. I run it in all my devices an I'm very happy.

I was told by the developer that he uses it also to maintain hundreds of machines which don't have a public IP.

well the idea is not to trust any company or service or even reveal my password/key to it. my use case was only escaping the restricted wifi, not managing machines.

This is just one TCP connection though, not like a connection to the Internet which you can use to make various TCP connections.

You can pass PPP or SLIP via WebSockets and connect to the Internet that way. That's already used for connecting DOSBox running in a web browser to the Internet: http://blog.vrcade.io/2017/03/setting-up-a-visp-using-pppow/

Before I ask my question, let me ask a "am i right or wrong."

WebSockets is a small layer on top of HTTP. And HTTP is a protocol on top of TCP. Right or wrong?

So then why would do this? To get TCP running over port 80 to get through a firewall? Why not just do TCP over port 80?

Like this project https://github.com/jpillora/chisel

Edit: above project is also using websockets. okay, i think i understand why you would do this.

My understanding is it's because there's no API for dealing with raw sockets in JS. Or rather, there is, and it's called WebSockets. (Depending on how you want to look at it...)

Well this thing is written in js, and needs to use raw sockets in its implementation. See "tcpSocket" and "pipe" in the code.

Sure browsers don't have a js raw socket API, but no browsers are involved in this thing.

Nope, there is actually a core API for dealing with raw sockets in NodeJS : https://nodejs.org/api/net.html#net_class_net_socket

Pretty straightforward to use, at that :

  const net = require('net');
  const client = new net.Socket();
  client.connect(port, host, function() {
    client.write("hello !");
  });

Interesting, thanks! So what is the benefit of WebSockets?!

In general: That you can use them from the browser.

For this use case: That they are an extension of HTTP, and thus can pass modern HTTP proxies.

I meant for this use case, i.e. when you have both available. Cool, thanks!

What this Node module offers is not raw sockets - let's not spread misinformation. ;)

https://en.m.wikipedia.org/wiki/Raw_socket

It does not even offer real access to a TCP socket, for example you can't set any options or flags in the TCP header of a packet. All that it provides is a way to open a socket and send / receive data. Very useful, but not a raw socket.

I stand corrected. Smarter every day they say !

yup, the core `socket` module in Node.js is meant for that.

Wrong again - the socket module does not give you raw sockets. If you were loo lazy to read the Wikipedia link I suggest you reconsider and educate yourself on the subject.

you're right, i was wrong, there is no core `socket` module (anymore?). but you were not right either.

node core can only tell if a file is a socket, but not read from it, at least that's what the docs say.

for people interested in this: https://github.com/santigimeno/node-unix-stream

Wrong again. What I said was that there was no native module in Node that would give you _raw socket_ access. And there isn't. Period. The thing you linked to doesn't either. Jeez ... :(

#1 reason is browsers. They expose HTTP and Websocket abilities, but nothing beneath that.

If, say, I wanted to connect to a (probably read-only) public database from my browser, I could not.

This isn't a surprise to anyone, of course. By most definitions, Web == HTML over HTTP.

> #1 reason is browsers. They expose HTTP and Websocket abilities, but nothing beneath that.

More specifically, web pages. Browsers usually support more protocols (FTP, for instance) but don't expose the protocols to scripts running on web pages.

WebSockets is not really a layer over HTTP as such; it uses an HTTP request format to set up the connection, but then it's mostly just a raw socket (with message framing, like in UDP).

Your question is still valid, I'm just dispelling the myth that WS is like TCP-over-DNS or something similarly inefficient.

It's almost exactly like TCP-over-DNS. Both are stateful protocols on top of stateless ones. The only difference is HTTP responses can essentially go on forever, whereas DNS ones are typically UDP with an upper bound on size, and TCP ones have a timeout on the order of seconds, and thus need to re-encapsulate further messages.

The biggest difference there is that while TCP-over-DNS just has TCP->DNS->UDP as overhead, the WS method has TCP->WS->TCP. Because the connection is stateful and the WS encapsulation is minimal, it's more efficient (and HTTP tunneling is almost always more functional than DNS or ICMP, anyway)

It's still inefficient. And it's not a raw socket, it's an application layer socket, essentially.

Are you talking about WS itself, or this tunnel thing? Because I was just talking about WS itself, which doesn't have any TCP over it, it's just framed data over regular TCP.

Anyway, looking at the code of this tool, it doesn't seem like there's any actual TCP being tunneled; from what I can tell, the raw TCP terminates in the application, and the data is piped directly into the WS connection. That's why you have to choose a static target, whereas with a TCP tunneling system like iodine, you can use it as an open gateway.

Fair enough on the raw socket, I meant a TCP socket.

I kind of jumbled the two things together in that comment, but WS itself is basically a stateful UDP over HTTP, so i'll compare it to that. (TCP has too many features to be compared to WS)

Comparing it to stateful UDP over DNS is a totally fair thing because in practice that is how you use it. You steal the ports of a completely different application to send data for your own different application, and do the hokey-pokey of one protocol before you then encode a payload in a completely different protocol before sending and after receiving it, and the application protocols it abuses to tunnel its data aren't even of the same state or connection mode.

Saying WS is "just framed data over regular TCP" is like saying stateful UDP over DNS is "just framed data over regular UDP". The only difference is that DNS has message reply limits. If you could keep sending one long DNS response payload, WS would be virtually identical to stateful UDP over DNS (when using tcp for the DNS).

And yeah you're right, this app is not encapsulating TCP packets, but it tunnels applications which use TCP. A lot of tunnels do this, although I don't think they advertise themselves as "tunneling TCP through X".

Because of the nature of HTTPS, combined with Websockets, would give you a really easy way to proxy yourself out of restrictive internet situations. Use an HTTPS connection to Google to domain front to an App Engine instance that has this running on it. Now you can TCP to anywhere from anywhere and it just looks like HTTPS traffic to Google.

if i understand you correctly, you want to set up the tunnelling server on the App Engine instance and the client on your local machine, right?

in this case, accessing google over HTTPS locally would not work properly, as the `Origin` header as well as the SSL cert wouldn't match.

There's no need for an Origin header and the SSL cert would match. That's the whole premise of domain fronting. You send the request to whatever IP resloves as Google.com, but when you make the request, your Host header is for the AppEngine instance. Because of the way Google (and AWS and I'm sure many others) route requests, they don't care that you're using the cert for Google.com, they just happily route you to whatever Host you specify. You may have to spoof some headers to do the 101 Switching Protocol correctly, but you have all the information you need to do it.

thanks for mentioning chisel!

Do not mute errors. It is a bad practice.

https://github.com/derhuerst/tcp-over-websockets/blob/master...

i'm aware that it usually is a bad practice. but mostly it is about handling errors as a valid state of the application, whereas in this case i'm not interested in any connection errors whatsoever.

Instead of stuff inside websockets inside TLS, wouldn't it be more efficient to just do stuff inside TLS?

It's encrypted so firewalls shouldn't be able to tell the difference.

you'd be surprised what a "deep packet inspection" (better known as DPI or L7-aware) firewall can tell. some are quite sophisticated, and there are a surprising number of techniques to identify even encrypted traffic (not the precise payload, per se)

Well there's ALPN/NPN, but you could lie about those.

I would indeed find it surprising if any commercial firewalls distinguish websockets over TLS versus raw bytes over TLS. Not that it's impossible, but I wouldn't think it would be done commercially.

Firewall packet inspection is basically just unpacking protocols layer by layer. Once you find the TCP or HTTP layer, you see if what follows looks like a WebSocket, which has a unique fingerprint and predictable elements. You can also look at streams and find patterns, like how one packet looks like HTTP, and then the following ones on the same stream don't look like HTTP. Initial versions may not be very accurate, but on "extremely paranoid" settings you can simply reject traffic that seems suspicious.

Also, having the DoD as a client will buy you some serious R&D time.

You can't unpack protocols layer by layer when they're encrypted.

True, you would typically use machine learning, which can be used to identify and separate layers in protocols.

Machine learning on random data won't get you anything.

And is machine learning fast enough for live network filters?

It was fast enough 10 years ago. And it worked on random data. Luckily though, internet traffic is not random data, and encrypted internet traffic is much less random than you'd think. Of course a lot of it involves inferring information about different parts of flows, but high accuracy is not difficult. (What's difficult to detect is traffic you haven't got any trained models for)

If it's actually random data then no it won't work. Actual random data is 100% noise and 0% signal. If the ML thinks it's picked up some signal, then it's wrong.

You're right that TLS is not 100% random, there is some information to work with: the size of the encrypted blobs (they're rounded because of padding) and the timing of them. Are there any commercial firewalls that do analysis of this kind for the purpose of blocking the traffic? It wouldn't be able the block all the traffic, because it would have to wait a while to get more timing data that it can apply its heuristics, then end the connection.

And it wouldn't be able to unpack protocol layers.

You're right, I misspoke, "random" data is nothing, but "random" as in "arbitrary selections of network data" it will work on. And no, it doesn't unpack layers per-se, but it detects signatures and can be used to find patterns and eventually separate layers of protocols.

The two oldest and most successful methods that I know of are matching on payload length and models trained on the initialization of network application protocols, neither of which requires constantly re-sampling and re-classifying to get a hit. And blocking traffic is often more about terminating an existing connection once you detect something bad going over it (deep content inspection).

Yes, commercial firewalls do look for tunneled applications. Palo Alto Networks has a patent on it (App-ID), Websense/Forcepoint does it (Content Gateway Analysis), Cisco sort of implements it (Network Based Application Recognition/Application Visibility and Control).

A bunch of open source software implements it, too. Some commercial proprietary software is also out there, with PACE leading the pack. Wikileaks has one of their product data sheets: https://wikileaks.org/spyfiles/docs/IPOQUE-PACEProtAppl-en.p...

Honestly, a lot of customers simply force proxies on their users and inspect all their traffic and drop anything that it can't inspect, so there probably isn't a lot of commercial need for this. But it is out there.

Random data, but what about traffic patterns and timings? I am pretty sure DPI engines are not just based on the packet's data anymore.

Corp firewalls often only ever see HTTP due to their CA certificate being installed on every machine and refusing "anonymous" HTTPS.

It doesn't mention whether the client verifies the TLS certificate. If not, that's bad. If it does, then this would have to run on a computer with the corporate CA trusted. It might not be a good idea to run this thing on a computer you don't own on such a restricted network, due to other monitoring on the computer itself.

For sure. Corp desktops often have their running programs surveyed. That said those more likely to run this program (engineers and IT folks) tend to run a lot of esoteric software compared to to the typical desktop. A weirdo or two just fall off the edges of monitoring.

Not related to the post, just a general question about WebSockets: can someone explain why long-polling was insufficient?

At a minimum, long-polling requires a server round-trip per message. Among other problems, this limits throughput to (message size / latency).

> At a minimum, long-polling requires a server round-trip per message. Among other problems, this limits throughput to (message size / latency).

I don't see why? Just have N multiple outstanding requests to the server (with N being large enough to achieve whatever communication rate you can accept). When the server wants to send you something, it picks one and replies to it. While you're busy processing it the server can still reply to anther one and send more data. When you're done you send more outstanding requests to the server to reach N again.

And if you ever run out of outstanding requests, then that's equivalent to your socket not being able to accept a new connection because there are too many pending already, which can always happen.

Yeah, you could have multiple requests hanging open with the server to enable faster server sends (and if a client wants to send data rapidly it could make multiple parallel POST requests).

The downside is you end up with a complex stateful protocol, with multiple sockets, and need the ability to handle out-of-order data. If you're targeting a platform that can't do HTTP streaming, then sure, but otherwise this is a pretty gnarly road to go down.

My personal preference for long-polling is to use it for stateless / RESTful APIs that have infrequent updates. Super clean, easy to use from the client's perspective, and efficient enough. I would not use it for stateful communication (IMO SockJS and Engine.IO are gross but I understand why they exist).

WebSockets are more efficient for rapid bi-directional communication.

But if this is not needed, then long-polling is perfectly sufficient for push delivery.

I see. What exactly is the step that is the biggest bottleneck in long polling that is being eliminated here?

Mainly avoiding the round trip needed for each payload sent by the server, as mentioned in one of the other replies. It also avoids this issue with data sent by the client.

Note that these things can also be solved with HTTP by using streaming, so WebSocket isn't strictly required as a solution. About the only difference between two single-directional HTTP streams and a WebSocket is that the WebSocket ensures data in both directions travels the same network path, which may make scaling or stateful sessions easier to build.

Enjoy! :)

https://tools.ietf.org/id/draft-loreto-http-bidirectional-07...

Just realized I missed your comment. Thanks for the link!

Setting a wss:// URL as homepage was a clever idea, but GitHub truncated it without making it clickable. :-\

TCP over web sockets over TCP.

Sigh. Of course I develop network virtualization software so I'm just as guilty of it.

Big fan of ZeroTier here :)

(I also run a semi-related project, without the decentralized magic of ZeroTier)

Actually TCP over web sockets [over TLS] over TCP.

How's your buffer bloat? — Fine, how's yours? — Great, thanks for asking!

How is this any better than the plethora of HTTPS VPNs which have been around for nearly two decades?

Apples and oranges. This tool is a proxy not a VPN.

Not really better, just an alternative. Chisel is more than likely much faster anyway and does the same thing.

HTTPS VPNs require TLS, and some proxies don't let you use TLS, because you might tunnel with it. It's pretty easy to block most HTTP tunneling because they typically use the CONNECT method. Ones that simply do an HTTP handshake and then pass the socket to a tunneling program can be detected or simply terminated after the session has gone on for a while or transferred a certain number of bytes. BOSH connections are a little more reliable, and I don't know off the top of my head if any tunneling apps support chunked encoding/multipart/etc, but it's possible to implement a tunnel using only valid HTTP methods, but it would be inefficient. Basically, most tunnels can be defeated without negative impact to regular users because HTTP isn't supposed to be used that way.

In comes WebSockets. Since it's a part of the modern web, and disabling them would negatively affect user experiences on the web, you can abuse them to tunnel random crap and proxies can't really do anything about it. The protocol even allows for obfuscation to make it difficult to see what's going over the socket (though it was intended to prevent cache poisoning attacks).

So in theory, if you can reach a target HTTP server, WebSockets may be the most reliable method to tunnel a connection over HTTP. The fact that it also supports multiple streams may make it faster than alternatives. And the fact that you don't need to use TLS may make it even faster, assuming the app you're tunneling does its own encryption.

My first thought when I saw this was "Hey! I made one of these". Turns out it goes the other way. From websocket to TCP. The thing I made was something where you connect to a socket(typically a unix domain socket in /tmp) and it connects to a web page via websockets.

I see some potential to work around corporate firewalls.

If you had made this comment to the introduction of HTTP itself, I would today be hailing you as a visionary

Woohoo! Someone built TCP

built on WebSockets

built on HTTP

built on TCP

This is like where newborn babies and the elderly have a lot in common.

WebSockets aren't built on top of HTTP. The initial handshake resembles that of HTTP, but once the connection is upgraded, all compatibility is lost.

In other words, WebSockets is a TCP protocol that mimicks HTTP during the handshake in order to be left alone by proxies and whatnot.

(edit) WebSockets isn't built on top of HTTP, but it is effectively an extension of HTTP. Not officially of course - then they would have had to get people to agree to an extension, so instead they called it a new protocol. But basically it's an HTTP request that turns into an obfuscated socket.

When WebSockets is negotiated on an HTTP connection, it ceases to be an HTTP connection; there is no way back to HTTP without closing the connection. It's kind of like when you start ppp, slip, or slirp on a shell you dialed into -- it's no longer a shell with a virtual terminal, it's changed into an internet connection, and you (usually) can't go back without dropping carrier and reconnecting.

There is really no such thing as an HTTP connection. There are request-response transactions, and these can happen without needing to break down the stateful connection they're transmitted over.

Ahem: https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_tunne...

  In this mechanism, the client asks an HTTP Proxy server to forward the TCP
  connection to the desired destination. The server then proceeds to make
  the connection on behalf of the client. Once the connection has been established
  by the server, the Proxy server continues to proxy the TCP stream to and from the
  client. Note that only the initial connection request is HTTP - after that,
  the server simply proxies the established TCP connection.

Both HTTP CONNECT and WS are designed to work over http proxies, both are designed to tunnel arbitrary application data, both use http to initiate a connection, both use http application ports for these connections, and neither of them return to http request-responses once they're set up.

But somehow, WS is a separate protocol and stops being http, while HTTP CONNECT is just an HTTP extension and becomes some other application proxied over http.

I guess WS is just special.

Reply on news.ycombinator.com