Socat – A utility for data transfer between two addresses (2018) | Hacker News Reader

Socat – A utility for data transfer between two addresses (2018)

copyconstruct.medium.com

230 pointsbrigandish65 comments

I'll join the chorus praising socat, but also share my gratitude for Gerhard, the developer / maintainer.

We exchanged emails back and forth about a use case I had, which was to split a serial device into two PTYs, allowing a process to control one while I monitor the other with picocom. We couldn't get it to work, so I gave up, but then Gerhard wrote back a few months later saying he added it as a feature.

It can be frustrating to figure out the right incantation for some advanced use cases, but this is more a complaint about esoteric corners of the operating system than socat which exposes the full complexity to you.

One of the invocations I recently worked out was to wrap an interactive process with a PTY, so that I could make a mock serial device that emulates a piece of hardware.

    socat PTY,link=/tmp/dev,rawer,wait-slave \
      EXEC:"./emulator",pty,setsid,ctty,echo=0

I use socat daily.

Nearly all of the microservices we develop have a way to be run locally with all dependent services using docker-compose.

The problem is most teams don't provide a way to do efficient local development for their particular service. Some do remote debugging into the built/running container (slow and requires rebuilding docker image for every change). Some inspect logs to debug (yikes!).

When I want to debug a particular service in IntelliJ, I simply remove it from the docker network and replace it with a container running socat, forwarding the appropriate port to host.docker.internal (the host machine). This is on a MAC, of course, where Docker runs in a VM.

This allows me to do in-IDE debugging of any service that runs in the docker network without having to rebuild the image every time I make a change.

I know there are IntelliJ plugins that provide a similar workflow, but I believe they all build an image and deploy a container to the network for every change. I could be wrong.

Interesting .. Can you provide an example?

How do you swap out the container in docker-compose?

Do you just have a service defined for each, and only start one of them?

> remove it from the docker network and replace it with a container running socat, forwarding the appropriate port to host.docker.internal (the host machine

Would you happen to have a fully worked example of this? We to have some services that we can and do run via docker compose - but it's often a cumbersome process to move one and one service out of the docker compose setup for debug/devel.

I use these two socat commands for MITM all the time on my own stuff so I can see the traffic:

  socat  -v TCP-LISTEN:666,forever,reuseaddr,fork TCP:example.com:8080

  socat -v TCP-LISTEN:8080,forever,reuseaddr,fork ssl:example.com:443

Excellent article!

I’ve tried to grok socat a few times but gave up quickly upon being faced with its screenfuls of dense help text.

This post sheds light on the core concepts behind the tool in a way that the man page and help text just didn’t do for me.

Now that I ‘get’ it, I’m actually excited to find an opportunity to use socat! I can think of so many interesting use cases.

A similar tool: spiped[0]. Connects two arbitrary TCP4, TCP6, or UNIX socket addresses, encrypting + authenticating data using a preshared key. For many applications it's basically socat + security.

[0] https://www.tarsnap.com/spiped.html

Have they done any audits or had critical analysis of their security code? These days I'm super hesitant to consider anything that hasn't had a lot of eyes on it. It's just too easy for a tiny screwup in protocol design/implementation, random number generation, etc. that puts everything at risk. Why risk it when I can just use wireguard.

cperciva is the author of spiped and a security expert [0].

But your question about an audit is still a good one! Tarsnap has bug bounties, but the top hit for “spiped security audit” led to a sort of amusing flame war on HN [1].

[0] https://www.tarsnap.com/about.html

[1] https://news.ycombinator.com/item?id=7539903

Lots of people have looked at the spiped source code; it has been around since 2011 and I've even handed out a few bug bounties.

It's also considerably simpler than wireguard, which reduces the potential for vulnerabilities; and operates on a per-connection basis, which means you can do things like establishing a secure connection to another host but have a socket endpoint which can only be accessed by certain users.

"lots of people looking at source code" is contemporarily a weak argument against audit and review by experts, I really love spiped.

"lots of people looking at source code" is contemporarily a weak argument against audit and review by experts

I would absolutely love to get some deliberate code review from experts. I know tedu has looked at some of the code -- he sent me some minor corrections -- and I would certainly count him as an expert, but I don't know if his review was systematic or if he just glanced at a handful of files.

That's awesome, thanks for clarifying! Looks like a really handy and useful tool to keep in the toolbox.

Socat is the default implementation for Kubernetes port-forwarding [1] when using dockershim for the kubelet runtime. Similar to how tar is used as the default Kubernetes cp command implementation.

[1] https://github.com/kubernetes/kubernetes/blob/770d3f181c5d7e...

I often use a socat docker container to proxy network traffic to docker containers. It effectively allows creating dynamic docker port mappings.

For example, I have a database with no docker port mappings (only a web application container in the same docker network accesses it directly). I can use a socat container with port mappings to proxy network traffic to the database.

Interesting, what's the benefit over exposing the port through Docker?

Security

https://en.wikipedia.org/wiki/DMZ_(computing)

It is exposed through docker, just via a separate container. The reason for doing so is that a container can't have new port mappings added after it is created.

Using the socat container allows me to access a port on my original container that I wouldn't be able to access unless I recreated that container.

As for why the database didn't have port mappings originally, it is best not to allow network access to things unless it's necessary. It's the principle of least privilege.

I've used socat in the past to debug all kinds of odd network behavior. It helped me discover a networking issue with some code by comparing performance of piping a Unix socket over TCP using socat vs. the native TCP implementation, which turned out to be caused by an overly conservative default send buffer limit.

Shameless plug [1], socat combined with WireGuard

[1] https://github.com/lrvl/tunnel-wireguard-udp2tcp

socat is an invaluable tool. I first came about it at my first job, where we (just two clueless dudes) had to migrate the mostly undocumented infrastructure of about 15 machines at Digital Ocean to Linode, adding replication and failover on top of that (making it ~35 or so machines).

During that migration, we tried to minimize downtime, so we made sure that requests made to the old infrastructure were aptly forwarded to the new infrastructure wherever possible, until DNS properly propagated. I found out about socat and we went for it. It worked great, and I marveled at the simplicity of the solution!

Ever since then, I've used socat both as an emergency saviour and as an actual service. I seem to recall using it once to go around some docker bug or shenanigans.

This is a brilliant idea and I’ve never even considered doing that form of port forwarding.

I’ll have to remember it for the next time I migrate.

This tool is like netcat on steroids. Definitely part of my stable toolbelt.

Whenever someone mentions netcat or nc I always ask "which one?". There are at last count 3 different implementations, all subtly incompatible. Which is another reason why socat is better since there's only one to deal with.

How do you refer to each implementation & what are some of the subtle incompatibilities?

The original one is the "TCP/IP Swiss army knife" released in 1995:

https://nc110.sourceforge.io/ (original release: https://seclists.org/bugtraq/1995/Oct/28)

Although it is "freely given away to the Internet community" with "an obligation to give credit where due", at least OpenBSD and GNU have seen the need to write their own versions under their project licenses:

GNU netcat: http://netcat.sourceforge.net/

OpenBSD nc: https://man.openbsd.org/nc.1

(The OpenBSD version has been ported to at least FreeBSD and Apple Macintosh OS.)

All of them have the same basic telnet's `host port` syntax for outbound TCP connections, but annoyingly the syntax for opening a local listening TCP socket varies. Say, you want to open a TCP socket listening on port 1234 (local), and a confirmation when it is ready:

The original and GNU netcats: netcat -v -l -p 1234

BSD netcat: nc -v -l 1234

That makes 4 implementations, because nmap also has one (https://nmap.org/ncat/).

Dammit, they have of course gone with the BSD syntax for opening a local listen socket (ncat -l 1234):

https://nmap.org/book/ncat-man-examples.html

Means half of implemations use `-l -p 1234` and half `-l 1234`.

Does BSD netcat give an error if you include the -p?

Yes, the manual explicitly notes that -p can not be used with -l: https://man.openbsd.org/nc.1

And this is indeed enforced, see line 424: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/nc/netc...

However, at least Debian ships a ported version of BSD's netcat. They actually apply a patch to allow using -p and -s with -l for consistency with traditional netcat: https://sources.debian.org/patches/netcat-openbsd/1.217-3/us...

Indeed. There are examples for making local/remote shells. It can make a pty, use setsid() to be a daemon, reset the terminal state, then listen(), etc, all in a one liner. Or route a serial port over ip via ptys in raw mode, again, with a one-liner.

I use socat along with npiperelay to allow the ssh client in my WSL sessions to access the Windows ssh-agent.

Interesting! Do you mind elaborating a bit? I ssh into WSL for this reason.

Likely something like this https://github.com/rupor-github/wsl-ssh-agent#wsl-2-compatib...

Similarly, you can also do this with KeepassXC

I followed the directions at https://stuartleeks.com/posts/wsl-ssh-key-forward-to-windows... to get the combination of socat and npiperelay to work for this purpose. I did slightly deviate from his final script by using pgrep instead of the combination of ps and grep to detect if npiperelay was already running.

Edit: These directions are very similar to those linked in the comment this is in reply to and either will work.

I haven't tried on WSL2, but on WSL1 this script also lets you integrate with KeePass's KeeAgent:

https://gist.github.com/duebbert/4298b5f4eb7cc064b09e9d865dd...

In the past I also used https://github.com/benpye/wsl-ssh-pageant to go from WSL to Pageant (which I believe KeeAgent is also compatible with)

Just to throw another one of these bridge applications into the mix, here is one that allows Pageant apps to communicate with Windows' native ssh-agent.

https://github.com/ndbeals/winssh-pageant

I use both this and rupor's wsl-ssh-agent from the other thread extensively.

When I looked for a similar solution quite a while ago one didn't exist so I wrote my own. I have been meaning to clean the code up and post it for about 18 months now but have never gotten around to it. Need to check this out and see if it is worth still using what I wrote.

Let that be a lesson to not abandon your side projects :)

I was in need of a solution to this issue for a long time before ndbeals thankfully released this about 9mo ago. And I was checking around the web regularly! I nearly wrote my own as well.

The use case for me was to allow WinSCP and Sourcetree to be used with the Windows ssh agent.

WinSCP is the reason I wrote mine. I'll probably stick with the one I wrote since it is win32 native and seems to consume less resources. I'm not a developer by trade and barely one by hobby. This was the first and so far only win32 app I have written so the code while functional is very ugly. I'm sure anyone familiar with win32 programming would be horrified. I should just throw it over the wall as is. Obviously the cleanup I was going to do before releasing it is never going to happen.

I use socat a lot, especially when wanting to transfer files without incurring the ssh encryption overhead or rsync daemon setup. Just tar | scocat and voilà, Superfast whole directory tree transfer. Sprinkle a bit of mbuffer or openssl on top and you can do a lot!

It talks serial too. I've used socat to send out data to the network from a set of serial connected devices for environmental modelling. Ended up being one of those jobs I got done from end to end in an afternoon with the socat part taking 10mins

Oh! I might as well use it for line editing on serial line with the readline local and serial remote addresses, for example when talking AT to a modem. Great! :)

And with history and all that stuff, too! I guess it can be used to forward serial port over network too.

Is it possible to use Socat (or any utility) to redirect a UDP connection on a local Windows machine to TCP port 443/80 to a remote system listening on 443/80 that will connect to the original intended UDP connection?

I can also do port redirects with iptables that can assist with solving the problem.

I need to host a microservice that uses UDP which won’t connect out on most corporate networks due to firewalls. I need to get the connection out via a standard TCP port.

Nothing malicious. It’s a basic corporate client/server app, but it’s bound to a UDP port that I can’t change.

I think your best bet is to setup a simple WireGuard for this, also udp but put the server on 53 or something it’s unlikely to be blocked.

Oh or skip that abd just use pen ( http://siag.nu/pen/ ) to get the traffic from 53->whatever port (can’t remember if haproxy supports udp)

huh. I haven't heard of this one. Seems like it's been around a while? Why might I use this compared to something more well known? haproxy doesn't seem to do generic udp, but maybe linux virtual server or something like that?

shrug -- it's just something I've used to do it, if it's a single server then I guess a simple iptables rule would also do the trick :}

We use socat to buffer logs sent from containers (docker-compose) during the short window at startup where logstash and elasticsearch are not ready yet. Works great!

Have you happened to blog about this? I'd love to be able to fix our setups with this trick!

Similar: using it to get syslog out of self-chrooting ssh with minimal trauma: socat -u UNIX-RECV:/home/sftponly/user/dev/log,mode=666 UNIX-SENDTO:/dev/log

It's also an CLI IMAPS client, with history: socat READLINE,history=$HOME/.imaps_history EXEC:'"openssl s_client -connect mailserver:993"'

or a CLI web browser with history: socat -d -d READLINE,history=$HOME/.http_history TCP4:www.domain.com:www,crnl

or, if you have to interact with something that has no readline, say sendmail: socat READLINE EXEC:"sendmail -bt"

> if you have to interact with something that has no readline

very versatile - socat PTY,Link is awesome.

very underrated. I've used this trick before to mock serial ports on some Linux-running industrial devices so I could debug them.

I made a small utility that'll replay socat log files captured by the -lf -x -v flags with the timing it collects if used in this way. It is very simple and is basically only there if you can't be bothered to roll your own: https://gitlab.com/inivekin/socatplayer/-/releases

So if you're commonly doing a socat process like: socat -x OPEN:/dev/ttyS0,b115200,echo=0,icanon=0 PTY,link=/dev/ttyO0,rawer 2> socat_log.log

then it may be of interest to you.

Great article! I use socat all the time to test multicast subscriptions in Linux.

Do you have any links/writeups to how you do this?

Finally socat is getting the luv it deserves.

Socat is terrible.

When used to establish a persistent connection to a remote listening (TCP) port, it has NO WAY to detect that the remote end has dropped the connect, and automatically reconnect. It just does not work.

Yes, I know all about all the keep-alive and retry options in the manual. Try it yourself.

It's one of those horrible tools that you can get up and running quickly, but can never make work reliably.

i doubt that socat did not handle TCP RST. is not it the network environtment which eats TCP RST in some cases (like AWS NLB).

That's why I said try it yourself... It's no good for serious use. Here's the first search result:

https://stackoverflow.com/questions/9341254/reconnect-socat-...

I don't think you understand how TCP works. A TCP connection can stay alive through an unplugged Ethernet cable if it hasn't sent any packets.

I've worked as a senior network admin for some very large organizations. Safe to say I understand TCP.

Reply on news.ycombinator.com