Serve2d – A protocol detecting server
github.com
github.com
In the end we modified our clients to include a decoy cipher suite in the SSL negotiation. That kept the network happy, and was enough for our multiplexer to then internally route to the correct backend.
You can get the SSH client to connect over this either by using an openssl s_client trick, or by just using my little tunnel tool (https://github.com/joushou/tunnel).
frontend mux
mode tcp
bind EXTERNAL_IP_HERE:443
acl is_tls req.ssl_ver gt 3
acl is_ssh payload(0,8) -m str SSH-2.0-
tcp-request inspect-delay 5s
tcp-request content accept if is_ssh
tcp-request content accept if is_tls
use_backend ssh if is_ssh
use_backend https if is_tls
where 'ssh' and 'https' are tcp mode backends.First of all, I'd like to thank my wife, who helped motivate me, my parents, who you know, did the thing that brought me here, my work, for paying me to browse news and occasionally code...
So, why Go? Because it was easiest that way. Try to take a look at how simple the ProxyConn thing is, which is the core of the entire thing. Go is a tool that I found suitable for the task. It also brings things like easy cross-compilation after the code has been written (Seriously, GOOS=windows go build on my mac and I get a PE32+, GOOS=darwin go build on my linux desktop and I get a Mach-O binary). You also get to interact easily with the wonderful Go infrastructure, in case you don't want to redirect to external services.
Why make something that already exists? Well, I didn't know that there were so many solutions, I just thought "Hmm, I wonder if I can make this work in a nice fashion...". That's not going to stop me from continuing development of my own, however. I have ideas I want to try, features I want to implement. It's powering everything on my own servers, and the stealthiness of SSH over TLS (NOT just SSH on port 443) have been (ab)used multiple times already.
Is my solution better than the others? Well, I of course like my own solution, but with few exceptions, I think I bring some interesting flexibility to the game. A lot fo solutions seem to be fixed for a certain purpose, such as SSH and HTTPS, usually just in their regular variants. I don't really care what you want to serve, as long as you can write down some unique bytes in the config for serve2d, or write a more complicated handler directly for serve2. Add transports if you want. Want SSH over TLS over WebSocket over IP over avian carriers? Add the transports.
I'm going to sleep now, and I hope the world haven't self-ignited before I wake up. It's really exciting and motivating to see how people react to private projects.
Don't get me wrong, it's kinda cool, but it seems like a really bad idea to actually do it.
To bypass this, I wrote a simple ruby script to tunnel TCP connections, while adding fake HTTP headers to get through the firewall.
https://www.torproject.org/projects/obfsproxy.html.en
https://gitweb.torproject.org/pluggable-transports/obfsproxy...
OpenVPN can use TCP, it can use UDP, it can use whatever port you like, it can even use a fixed key to turn all traffic into random noise (obfuscating protocol structures).
Conservation of the port number space is required because this space is a limited resource, so applications are expected to participate in the traffic demultiplexing process where feasible. The port numbers are expected to encode as little information as possible that will still enable an application to perform further demultiplexing by itself. [...] IANA strives to assign only one assigned port number per service or application.
Why do I know this? Consul by default uses 5-6 ports, and one of them conflicts with an IETF registered port in-use by our data center provider. Was hoping that the RFCs gave some ammo for a resolution in my favor, came out more humble and wondering if Consul really needed that many ports.
Some of it understandable, e.g. when one port is for stuff that is always meant to stay behind your firewall, and another for public access. In that case it actually makes things easier to do properly. But so often it appears to be entirely gratuitous.
There are use cases - many nefarious cases but some real world cases.
Let's say your university allows everything through 80 but blocks everything else. You could SSH home or stream content.
Another use case could be that you have a web site and game server on same server. An interesting implementation could be to listen on port 80 so that they could connect to yourserver.com via HTTP and view information about the game/server and connect on the same port. Interesting but not very practical.
I suppose an acceptable/respectable use case would be custom load balancers or honey pots.
This advantages of serve2 is that it is 100% go and can be integrated and used without any extra dependencies or multi-setup :)
To me, an advantage in usability is having a PPA with properly built .deb packages. If I have to use a language-specific package manager that I don't already use regularly, you've likely lost me, unless I really need this functionality. If it doesn't come with a proper daemon mode (correct forking, PID file support, proper file or syslog logging), sample config file, man page, or an init file, that's even worse. I am much less likely to use this in any type of "production" environment if I have to maintain those pieces myself. Running things in a screen session is so "I'm running a Minecraft server".
That is not to criticize your work. You've done a great job! serve2d looks very interesting and I might actually have to give it a try sometime.
Slide 19 of this if you are interested: http://www.bitpim.org/papers/baypiggies/siframes.html
ie other languages may not work, is less tested, and not normally done.
Go 1.5 has mechanisms for dynamic linking, though.
As is no doubt obvious from my tone, I'm not buying it and am very happy to see signs of a pendulum-swing back toward static linking and monolithic executables.
I believe it's more similar to Database Normalization from RDBMS world than to anything else. And the most important objective of Database Normalization is considered "Freeing the database of modification anomalies".
My own experience with shared libraries is pretty positive. I have fixed OpenSSL vulnerabilities many times by just updating the OpenSSL library and restarting all services. Compared to my own experience with docker where after waiting for a few weeks I had to change my base images (as nobody was updating them) updating just a single shared library and having the vulnerability fixed is way easier!
This, of course, is true if those who maintain the software you use do care about backwards compatibility (which tends to be true for the "boring" stuff and false for the stuff considered "cool" - looking at you nodejs library developers who break my build at least once a month).
But for most libraries it may really be overkill.
What we lack in the unix world is a coherent division between what ought to be a very small, stable, well-understood set of fundamental system libraries suitable for dynamic linking and the vast array of utilities a developer might choose in order to get an app built without having to reinvent everything from scratch.
Upgrading libraries out from under a built, tested executable is not something we should be doing lightly, because there is no possible way to know in advance whether the apps depending on the library have succeeded in programming to the interface rather than the implementation.
- no more having to deal with dependencies at packaging time, which makes packagers' job simpler because all they have to care is the one and only standard way to retrieve dependencies and build the binary. (Much like the standard way of doing things in C would be ./configure && make && make install, with the added bonus point that the dependencies are also taken into account). This also means that there's a higher chance that the software will be packaged in the distribution of your choice, because the bar is lower
- no more having to deal with dependencies at runtime, because each binary has everything it needs inside of itself. In practice this means "scp as a deploying method". It's an even lower common denominator than packages.
> If it doesn't come with a proper daemon mode (correct forking, PID file support, proper file or syslog logging), sample config file, man page, or an init file, that's even worse.
This is orthogonal to the choice of programming language, though. On top of that, I believe the application shouldn't deal with forking, it's the job of your supervision system to deal with daemons. All an application has to do is log whatever happens on STDERR and let the system handle that.
Static linking and bundling of dependencies is a no-no in most distributions. If anything, the Go model is a headache for package maintainers to deal with.
So, it is the same that linking the libraries statically? C and C++ has done that like since forever.
Many times actually, I prefer to deploy just one file whenever I can.
> just to be able to produce binaries that work on every Linux
That's a Linux design/decision thing. Linux binary compatibility is ... well ... challenging. In Windows is not hard at all (Not sure how is it in MacOS since I have worked mostly for iOS in Apple's world).
As for logging, there are loads of logging libraries that support both stdout and file logging. My policy is to support both for my project and it has been almost no burden so far (in Python and in C). Not everything is containers, and having a feature like logging does not mean it cannot be used in a container.
Technically there's only one ssl library you should use, it's the standard one. This doesn't change your overall point that when a part of the program must be upgraded, the whole binary must be upgraded as well and re-deployed, which I totally agree with. If your software is a server that you host yourself and you have full control over the deployment chain, as is the mindset behind Go, then re-deploying a dependency or re-deploying a binary is more or less the same.
Regarding logging, I'm really partial to the approach advertised by 12 factors (http://12factor.net/logs): let your software handle the business, let the supervisor handle the software's lifecycle, and handle the logfiles outside of the software, because there are factors specific to the hosting machine that in my opinion shouldn't be the concern of the software.
Re-deploying the binary, and getting the updated binary are two different things. When OpenSSL has a vulnerability, they announce it after the fix is out, distro maintainers release updated packages, I run `apt-get update && apt-get upgrade` or equivalent. It is on OpenSSL and the distro maintainers to release the update, and on me to apply the update.
When we are talking about static linking, it's now suddenly on the software developer to release a new binary or on me to rebuild the binary I have from source. Now I have to keep track of (a) which dependencies each project uses, and (b) which vulnerabilities come out. Not being familiar with Go, does it have such dependency tracking framework where I can update all packages where dependencies have been updated? Of course once I know that I need to perform an update, it doesn't matter if I run `apt-get upgrade` or `go build foo`.
As for logging, I advocate doing both. I really should make a separate blog post about it, but here's what I expect a well-behaved daemon to do:
- Always start in foreground and log to stdout (otherwise it seems like it exited without any output)
- Use the -d and --daemon flags to go into background
- Use the -p and --pid options for specifying the PID file
- Use the -l and --log options for specifying the log file location (if not specified or is - use stdout)
- If uses a config file, use -c or --config for the location of the configuration file. Default to standard OS location.
This way all possible modes are supported (running under a supervisor process, in a container, as a stand-alone daemon, or in the foreground while in development/testing/experimentation), and it is very easy, even in C to write software that meets these requirements.
Dynamic linking is not synonymous with security or ease of use. It's known to significantly reduce performance, both when loading as well as runtime whenever an external symbol is used, and the memory savings aren't too great. One has to remember that static linking also means that unused symbols are left behind, as the compiler knows what is needed.
Ability to update components of an application is one of the ideas behind dynamic linking, but in practice it doesn't work that well, and often requires that the application is updated to link against a newer version, which of course can only happen when the distro updates. This also includes LTS distros and backports, which you either have to wait for or kill support for.
There's also a difference in how Go vs. things like C and C++ handles linking, due to Go actually knowing about multiple files, and who uses what. This is quite a bit different than the copy-pasta that C preprocessors generate. We just ported a major product from C++ to Go, which gave us significant performance boost with considerably less code and complexity (This is not a "Go is better than C++", Go just provided a lot of things we needed in the std library that were hard to get in C++). The "necessities" were dynamically linked (libstdc++, libgcc_s, pthread, libc, ...), but our own libs were statically linked in. The binary was 51MB. The equivalent, completely statically linked Go binary is 8MB. It also does a clean build in <500ms, rather than the 3-5 minutes it took for the C++ project.
... But do remember, that Go 1.5 brings dynamic linking for the ones that want it. While the Go creators don't seem super fond of dynamic linking, they are providing it.
Regarding updating, "go get -u packagename" will update all dependencies, assuming the dependencies are go-gettable. Vendoring changes the picture a bit, in the sense that applications will bundle their own version of things to simplify things a bit, but that doesn't really change the go get -u command. Follow with a go build, and your application is up to date.
My rant is mainly triggered by dynamic linking being the standard without many people questioning the usability. It rarely it works as intended, especially with versioned symbols. If I depend on openssl, and an update comes in, then one of 3 things can have happened: 1. They updated the old symbols 2. They implemented new symbols, but left the old ones behind 3. They implemented new symbols, killing the old ones.
1. means that the behaviour of the library under my application changes, which can lead to unexpected bugs. 2. means that my application is not experiencing the fix at all. 3. means that my application won't start anymore, due to dyld errors. 3 is what happens when you update something to a new version in the normal manner.
This "multi-version" mess also makes the libs more complicated than they're supposed to be. My ubuntu libc, for example, includes symbols from 2.2.5, 2.3, 2.3.2, 2.4, 2.5, 2.7, 2.8, 2.11, and 2.15, just to check the very last symbols. It's a mess.
For users, it's mainly a headache when trying to get newer versions of packages that depend on newer libs. This isn't much of an issue if you're, say, a gentoo or arch linux user, but if you're maintaining Debian systems, and need a package from a newer repo but can't/don't want to dist-upgrade to testing/experimental, then you're practically screwed short of compiling things yourself.
For distro maintainers, it's a mess as all packages depending on the lib for that distro release needs to be recompiled when releasing a new version of the lib, which is a lot of work.
Dynamic linking and versioned symbols is also the very reason that causes sites with binary releases to have a binary for Windows, a binary for OS X, a binary for Ubuntu 14.04, a binary for Red hat 6.3, a binary for Arch Linux, a binary for..., further increasing the inconvenience for the user.
The only time you have benefits from dynamic linking is in the very rare scenario 1 of updated libraries, where everything is done exactly right when modifying old symbols so nothing breaks, which is a bit unlikely unless the bug fix was very simple. It also has to be serious enough that the library maintainers see the need to backport the bugfix to the old library versions, rather than release a new one. Otherwise, it's only dragging things down, both in performance, resource consumption and maintenance overhead.
Daemonization is at least a fairly standard feature, but with logging there's so many people with such varied concerns that getting fancy, trying to meet people's many needs, can lead to a lot of program bloat very quickly. Instead of going at these on a case-by-case basis, and now that we are more container-centric, it makes sense to run in the foreground and put your output on stdout, let the rest of the system support that utterly uncomplex pattern.
For things that become part of the OS, yes, I'd rather they come via some install approach that includes the necessary integration. But for anything else, I think a lot of our packaging approaches are dedicated to saving disk space and RAM, which is something that matters way less to me now than it did 15-20 years ago when CPAN and APT were designed. In 2000, disk prices were circa $10/GB [1]; now we're looking at $0.50/GB of zippy SSD [2] or $0.03/GB of spinning rust [3]. RAM is similarly about 2 orders of magnitude cheaper. [4] Given that, it makes a lot more sense to burn space to minimize the chance of a library version conflict or other packaging issue.
Another thing that has changed greatly is the pace of updates. 15-20 years ago, weekly releases sounded impossible to most. Now it's common, and some places are releasing hourly or faster. [5] Thanks to things like GitHub, the whole notion of a release is getting hazy: I see plenty of things where you just install from the latest; every merge to master is in effect a new release.
Given that, I think both Go and Docker are pioneering approaches that are much more in sync with the current computing environment. I'm excited to see where they get to.
[1] http://www.mkomo.com/cost-per-gigabyte-update
[2] http://techreport.com/review/27824/crucial-bx100-and-mx200-s...
[3]http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&Is...
[4] http://www.jcmit.com/mem2015.htm
[5] www.slideshare.net/InfoQ/managing-experimentation-in-a-continuously-deployed-environment slide 27
Go has advantage over scripting in speed, over C/C++ in memory management, over strict FP languages in popularity, and over Rust in being stable and known for longer.
Great! This seems to be a very good Selling Point for go!
If anyone's interested, Corkscrew is an alternative solution, allows you to disguise SSH as HTTP/HTTPS traffic, useful for getting through restrictive firewalls to administer remote machines:
I do think serve2d is considerably more flexible than Corkscrew, though, being able to tunnel anything over TLS if needed. Or other transports, if more are added. I also let you use the same ports for other things.
Postgres (which I would recommend over MySQL any day) seems to have a saner client-initiated protocol. I only read part of the spec, but I'll try to see what pattern would need matching later today.