Turn any application that uses stdin/stdout into a WebSocket server
github.com
github.com
http://www.mail-archive.com/fossil-users@lists.fossil-scm.or...
tl;dr: sqlite.org and fossil-scm.org run using a simple HTTP server via inetd and some parts of the website are CGI scripts. In 2010 they served over a quarter million requests per day on a tiny VPS with only 3% CPU load.
"Oh, this is a toy. Oh, this doesn't meet my Enterprise needs. Oh, this isn't EAL7¹ Certified, I can't run my Atombomb defense system with it."
Dear Ladies and Gentlemen, you have to admit that this is ueber useful for so many scenarios, that are out of the box, you're thinking in.
I can now hack a firebase clone, just for fun and my prototypes don't require the setup of dozens or a hundred npm packages. Besides the coolness of NodeJS, there are still some use-cases for hackers with this.
--
1. http://en.wikipedia.org/wiki/Evaluation_Assurance_Level#EAL7...
With this, you have to handle all of that in your application binary. You can certainly do this. However, if you are asking why you need to do this, you are almost certainly not ready to do this.
I agree that this is a pretty useful method of tying software systems together. However, one of the the systems you're tying together is the public Internet, where all sorts of danger lurks. Writing secure parsers is hard. I've written an HTML parser [1], one of the most well-tested ones out there (I spent the better part of a year doing nothing but testing & debugging, and ran literally billions of documents through it), and Google's security team still found 2 bugs in it with about a day's work.
Just as an illustration, from the linked page:
"Security. Gumbo was initially designed for a product that worked with trusted input files only. We're working to harden this and make sure that it behaves as expected even on malicious input, but for now, Gumbo should only be run on trusted input or within a sandbox."
If you have corrections, post them, which you did up above. My post is being upvoted because it makes people think about considerations that they didn't think about before, which is a good thing. Further details are also a good thing, but I've found - through numerous engineering projects - that if you think you've found the "right" solution and someone is just "wrong", there's usually a trade-off you missed somewhere and a concern they're considering that you're not.
Anyway, I don't think that your conclusion ("Security has nothing to do with using STDIO and everything to do with how you code your program") and my conclusion ("You can do this, but you better consider the attack surface you're opening your program up to first") are all that incompatible. My point is that most programs with an interface on STDIN are not designed out of the box to be run on untrusted, potentially malicious input, and so you will have to spend time hardening yours that far exceeds the time you spend hooking up the network protocol to STDIN.
This table is not complete:
http://bstring.sourceforge.net/features.html
.. and the thing is, its not the 90's any more. There are a lot of great tools that can be exposed on this interface, if you think about it, quite safely ..
Yes, that is a valid point. However that is not what you said, which was:
> Problem is that it requires that the app handle all the security aspects of talking to the Internet
which is not unique to applications using this daemon, or even to WebSocket applications in general. That's borderline FUD. Combined with the inaccurate statements you made to back it up, your original post is simply incorrect and misleading.
> if you think you've found the "right" solution and someone is just "wrong", there's usually a trade-off you missed somewhere and a concern they're considering that you're not.
No, not if the "right" solution is backed by facts and research, and the "wrong" solution is based on demonstrably incorrect half-remembered mistruths from ten years ago. There's not really a tradeoff there.
> STDIN/STDOUT is an unrestricted protocol.
So are WebSockets.
> Normally when you write a webapp you have a well-tested parser to validate the HTTP request for you
The only HTTP in WebSockets is the handshake, which is what this daemon handles for you.
> and you have safe APIs to access parameters,
There are no "parameters"; WebSockets transport a stream of opaque packets.
> and your templating language auto-escapes dangerous characters for you,
There are no "unsafe characters" in a binary protocol. At best you could claim NUL and LF, the former because it has special meaning to C, and the latter because it's used as a message delimiter by this daemon. The worst either of those will cause is reading less of a message instead of more.
> and the response goes through middleware to prevent XSRF attacks
You have to do this in any language you use. This has nothing to do with communicating via Unix pipes.
> With this, you have to handle all of that in your application binary.
Most HTML parsing engines are available as libraries.
It's oftentimes really dangerous to expose the raw stdin/stdout to the web. The point is that this by default has none of the usual safeties associated with making a program web-accessible. It's dangerous.
This doesn't make it a bad tool. It just means you need to be aware of the downsides.
Except he didn't, he said, "Problem is that it requires that the app handle all the security aspects of talking to the Internet", which is not unique to applications using this daemon, or even to applications using WebSockets.
But the parent says that "most command line tools were never written to be exposed to the internet at large", and tcpserver is a command line tool that runs other command line tools, so is tcpserver OK to use for "real work" by his standards? Maybe we should use a GUI tool "written to be exposed to the internet at large"?
http://en.wikipedia.org/wiki/Inetd
I actually don't mean to be snarky, this program acknowledges its heritage and is a cool hack!
If you use this on top of programs like bash, well ermm, you get what you deserve ;).
Here's an example of how I used websocketd to create a little dashboard for monitoring Linux CPU/memory/IO stats. It basically uses websocketd to stream the output of vmstat to a web-page that plots the numbers: https://github.com/joewalnes/web-vmstats
Other useful examples: tailing log files, executing long running job and monitoring output, or interactive querying of datasets that require a long running 'cursor'.
This is not for everyone or everything. Remember that like CGI, a process is forked for each connection so it's not the kind of thing if you want to handle a million concurrent connections on a single server.
However for dashboards, admin tools, quick mashups, visualizations, etc - it's a pretty handy tool.
{ Server } = require 'ws'
{ spawn } = require 'child_process'
command = process.argv[2]
args = process.argv[3..]
wss = new Server port: 8080
wss.on 'connection', (ws) ->
ps = spawn command, args
ps.stdout.on 'data', (data) -> ws.send data.toString()
ws.on 'message', (data) -> ps.stdin.write data.toString()
ws.on 'close', -> ps.kill()
ps.on 'close', -> ws.close()
ws.on 'close', -> ps.kill()
ws.on('close', function() {
return ps.kill();
});Edit: it was meant to be a reply to the parent.
ps.stdout.on 'data', (data) -> ws.send data.toString()
ws.on 'message', (data) -> ps.stdin.write data.toString()
ps.stdout.pipe ws
ws.pipe ps.stdin
I still have a question for Node.js experts... would the 'close' events had to be handled then? Streams handle and propagate the 'end' event just fine, but should the underlying resource be closed? I guess killing `ps` is mandatory since ending `stdout` won't kill the process (or will it?), but is closing `ws` still mandatory too?
This all provided you use a streaming web socket module like https://github.com/maxogden/websocket-stream
wss = new Server port: 8080
wss.on 'connection', (ws) ->The reason I switched to Go was to make distribution/installation for end users easier.
huhtenberg: You're right. It's very similar to inetd - that's why the first sentence in the websocketd description is "Like inetd, but for WebSockets".
The thing it adds on top of inetd is it takes care of the protocol so the underlying app doesn't need to. Namely HTTP WebSocket handshaking and data framing.
Huh? There are no version compatibility issues with Go. Just use the most recent version of the compiler to compiler your code.
Anything written since Go 1.0 is compatible with all the 1.x compilers. Anything written before that (which is truly ancient code at this point) can be updated automatically with the 'go fix' command.
# Self contained Go build file that will download and install (locally) the correct
# version of Go, and build our programs. Go does not need to be installed on the
# system (and if it already is, it will be ignored).
1. So developers can do a clean checkout and it will just work. No external dependencies needed (e.g. manually installing Go, setting GOPATH etc).
2. To ensure that anyone who builds and releases it always uses the exact same version of Go.
Please don't do this! It's attempting to solve a problem that doesn't exist.
As I mentioned above, Go does not have compatibility issues with different version of the compiler. This is a huge strength of Go, and spreading the idea that using the same version of Go is somehow beneficial creates misunderstanding for new developers.
I challenge anyone to find an example of Go 1.x code in which using the most recent version of the compiler will build an incorrect binary. Seriously, if you find one, please let me (or better yet, the Go mailing list) know, because the Go team has made a dedicated effort to make sure that this doesn't happen. (Such cases exist, but I am fairly certain without even looking that this program doesn't run into any, whereas it's clear (below) that the Makefile for this project doesn't handle cases that the standard build tools do.)
I didn't understand this when I started using Go over a year ago - at the time, I also used Makefiles over Go's built-in build tools, and did a number of other non-idiomatic things surrounding the build process. More experienced Go programmers told me I was wrong, and I didn't listen at first. In the end, I finally realized that they were right: it added complexity for literally zero value. Please don't make the same mistakes that I did.
With the Makefile I had to alter the static Go version to "darwin" to get it to pull the correct source.
While using older versions of Go is not recommended, it's not going to cause you any problems unless you're relying on (e.g.) 1.2-specific syntax like the three-argument slice syntax introduced in 1.2. If you are doing that, then older versions of the compiler will fail to parse the code, which would hopefully motivate the developer to upgrade their system-wide Go installation[0]. That way, they still only have one version of the compiler installed, and it will work for all programs they need to compile.
The only people who need to have multiple different versions of the gc compiler installed on their system are people who know what they're doing and handle this themselves manually (ie, the core contributors to Go).
One of the explicit design goals of Go is to simplify the build process. A side-effect of this is that the build process and best practices are different for Go than for most other languages.
I appreciate the effort that you've put into trying to make this easy to deploy, but using Makefiles and local versions of the compiler in this way is generally bad practice for experienced Go programmers, and encourages bad habits for inexperienced Go programmers.
[0] Ubuntu's repositories are hilariously behind the times on this. That's a separate matter of discussion, but it's why it's not encouraged to install Go through the main Ubuntu repositories.
websocketd --port=8080 bash
ws.send("ls")
ws.send('history');
ws.send('echo "' + pubkey + '" >> $HOME/.ssh/authorized_hosts');
To just open up the admin area and see what's going on, provided they're indeed full-permission admins. Then, to actually send input from said interface, that could make ssh'ing into the server something one need not do often beyond initially setting up the app.
Thanks for the idea. Assuming we're not running as root, and the admin side of things is secure, am I not considering any critical pitfalls of this approach? Also, any frameworky cmsy thingers that already do this? Is this new, am I a unique snowflake?
https://github.com/minikomi/pipesock
also in go.
Edit: Doesn't do any receiving, only pipes to a socket what it gets.
You have to be very conscious of input escaping and validation, though; otherwise your box will be owned in no time.
Like CGI, websocketd will pass additional URL path and query parameters as environment variables.
So you could access something like ws://somehost/?a=b and you'd access the QUERY_STRING environment variable in your script (which contains "a=b"). Virtually every scripting language already has a library for parsing CGI environment variables so you can do that.
Basically, websocketd tries to emulate CGI as much as possible, but provide WebSocket streams instead of HTTP responses.
Check your system time?