Show HN: A pure bash web server. No netcat, socat, etc. | Hacker News Reader

Show HN: A pure bash web server. No netcat, socat, etc.

github.com

224 pointsdzove85597 comments

Regarding the creation and removal of a tempfile (line 72 & 85):

    local tmpFile="${TMPDIR:-/tmp/bash-web-server.$$}"

    # ...

    rm "$tmpFile"

To avoid collisions when the PID is reused, and to clean up0 the tempfile on errors, I recommend using mktemp and trap:

    local tmpfile="$(mktemp --tmpdir="${TMPDIR:-/tmp}" bash-web-server.XXXXXX)"
    trap "rm -f \"${tmpfile}\"" RETURN EXIT

    # ...

    # Do nothing at the end of the function; trap will
    # remove the file at RETURN automatically.

Otherwise, I like the implementation! It's nice to see good bash techniques like parsing with "IFS='&' read -ra data" and rewriting variables with %%/etc.

I always use mktemp and trap to remove tmp files.

But in this script i would like to avoid external commands. I will even remove the use of tmp files kn thw future.

To spare you the trap you can open the file as FD in your bash process (e.g., exec {my_fd}>"$TMPFILE") and then directly delete it before doing anything else, and use the /proc handle to access it ("/proc/$$/fd/${my_fd}")

One more fun experiment: avoid the tempfile on disk/tmpfs by storing data into the pipe connecting two processes, where the first one can already exit for the pipe to give EOF when reading beyond the buffered data.

  true | sleep infinity &
  STDINPID="$!"
  TMPPIPE="/proc/$STDINPID/fd/0"
  echo hello > "$TMPPIPE"
  echo write more > "$TMPPIPE"
  cat "$TMPPIPE"
  echo write again > "$TMPPIPE"
  cat "$TMPPIPE"
  kill -9 "$STDINPID" # clean up pipe

Now I would like to avoid the additional process by using the current shell process instead.

Still needs to spawn a temporary process but it can be killed early:

  true | sleep infinity &
  STDINPID="$!"
  exec {mypipe}<"/proc/$STDINPID/fd/0" # hijack FD
  TMPPIPE="/proc/$$/fd/${mypipe}"
  disown "$STDINPID" # suppress killed message on stderr
  kill -9 "$STDINPID" # clean up process now already
  echo hello > "$TMPPIPE"
  echo write more > "$TMPPIPE"
  cat "$TMPPIPE"
  echo write again > "$TMPPIPE"
  cat "$TMPPIPE"

Since now, multiple concurrent connections will work after my patch. (Need to add subshell) i will need to consider using fd as output cache. Instead of temporary files.

This here is a simple echo server that uses tr to uppercase:

  exec {checkfd}>/dev/null
  CHECKFDPATH="/proc/$$/fd/${checkfd}"
  (while [ -e "$CHECKFDPATH" ]; do sleep 1; done) > >(true) &
  STDINPID="$!"
  WRITER="/proc/$STDINPID/fd/1"
  
  while IFS= read -r LINE; do
    # only echo lines if we didn't close the connection yet
    if [ -e "$CHECKFDPATH" ]; then
      echo "$LINE" | tr '[:lower:]' '[:upper:]' > "$WRITER"
      if [ "$LINE" = "bye" ]; then
        exec {checkfd}<&-
      fi
    else
      echo "received while closed: $LINE"
    fi
  done < <(nc -q 1 -l 8080 < "$WRITER")
  if [ -e "$CHECKFDPATH" ]; then
    # close checkfd to close writer pipe
    exec {checkfd}<&-
  fi

This variant here works a bit different:

  exec {checkfd}>/dev/null
  CHECKFDPATH="/proc/$$/fd/${checkfd}"
  (while [ -e "$CHECKFDPATH" ]; do sleep 1; done) > >(true) &
  STDINPID="$!"
  disown "$STDINPID"
  READER="/proc/$STDINPID/fd/1"
  
  { while IFS= read -r LINE; do
    echo "$LINE" | tr '[:lower:]' '[:upper:]'
    if [ "$LINE" = "bye" ]; then
      echo exiting > /dev/stderr
      break
    fi
  done < "$READER" ; kill -9 "$STDINPID" 2>/dev/null || true ; } | { nc -q 1 -l 8080 > "$READER" ; kill -9 "$STDINPID" 2>/dev/null || true ; }
  exec {checkfd}<&-

No need of tr here in your example.

The fd will only be used to stor it temporary since, we first need to sent headers, and the headers need to be sent before the bofy. The server base os working fine now with accept and a patch request os already sent to bash. So probaböy on the next release accept will be full usable.

Seems "sleep 60" is safer and still good enough unless the system is completely overloaded.

That's an awesome technique, thanks for it! It is Linux-specific though.

It's also useful for signaling between processes to have them continue doing something as long as this proc path exists (the fd can also just be backed with >/dev/null instead of a real file that needs to be removed)

Normally, the pid and fd could be reused, so you should either make it a known unique path and verify it's still the same, lock the pid via some means (not sure if and how it's possible), or use another technique.

For creating some kind IPC, i would create a named fifo or even use coproc (see man bash)

That's super clever, thanks for pointing this out!

can use $RANDOM to get more entropy/avoid collisions, though as you said, better not to use files at all.

My humble submission to this genre is this pipeline I wrote once, when I needed an HTTP proxy, but I couldn't use install a real one do to work limitations, and I couldn't use firewall rules because of weird limitations of the Mac OS at the time:

    echo "" > /tmp/buffer; tail -f /tmp/buffer | netcat httpbin.org 80 | netcat -lp 8080 > /tmp/buffer

I call it a circular pipeline.

I know this is a toy meant for short-term use, but you can use a fifo/named pipe so you don't have a growing file used for data transfer.

    mkfifo -m0600 /tmp/buffer && netcat httpbin.org 80 < /tmp/buffer | netcat -lp 8080 > /tmp/buffer

Might want the file if you want a record of course.

Thanks! TIL! That's a much cleaner way of doing it.

Regarding keeping the file, if memory serves the "real" pipeline had some `tee`s in it. I recreated this much more recently to make a GitHub repo of funny code snippets (https://github.com/MaxBondABE/one-thread-crashing/).

You reminded me of a different toy I made many years ago. I created FIFOs for files that you might try to read when exploiting an arbitrary file read in a web app, say /etc/www/apache.conf (if that's the right path - it's been a while since I've configured a web server!) and I'd have a program that opened them. This would block until someone opened the other end of the FIFO, at which point I'd raise an alarm (which was just a print statement).

That's awesome but would you mind explaining this a little bit more to those not very unix savvy? I think I get the first three parts:

1) Create a temp buffer file

2) Read the file constantly to STDOUT?

3) What does netcat httpbin.org 80 | netcat -lp 8080 do?

What would be a good use of this circular pipeline? Can you give me some example of what you use this for? Thanks

Of course!

A.)

Some context:

- Our goal here is to match up to two network connections, so that anything we receive on one is forwarded to the other.

- netcat is a "networking swiss army knife", it's a tool for making network connections

- HTTP and bash pipelines both happen to be "line oriented", meaning they operate one line at a time. Exploiting this happy coincedence is what makes this work

- httpbin.org is a metanym for any website you need.

So:

    echo "" > /tmp/buffer

Initialize our buffer to an empty state

    tail -f /tmp/buffer

Read each line in the buffer, one at a time, and feed them to the next command. Do this forever (-f).

    netcat httpbin.org 80

Connect to the website httpbin.org. Transmit each line from our buffer to the website.

    netcat -lp 8080 > /tmp/buffer

Accept a connection. Transmit any data we receive from httpbin.org to this client; write any data we receive from the client to the buffer, so that it can be transmitted to httpbin.org.

B.) I don't think there is one! I've never needed to do this again in the, gosh, seven intervening years. Normally bash pipelines operate like a bucket brigade, moving data unidirectionally through various processing steps, and usually dumping it into a file at the end.

They aren't normally for creating long-lived programs that manage bidirectional flows of data. Which is why writing a web server in bash is a fun challenge.

Thank you for the detailed explanation. Learned a thing or two too! :)

I'm not OP, but as they said, it's an HTTP proxy. Run it, then point your browser at http://localhost:8080. The message is proxied over to http://httpbin.org (on port 80, the normal HTTP port). If you're in an overly restricted environment and have a legit need to proxy to a very limited set of hosts, you can ad-hoc make a proxy mapping from local port to a single specific remote host. Just change the 8080, destination host, and buffer file.

And I mean this as encouragement, but you should read the synopsis from the GNU netcat manual. It's short and explains it well enough. netcat httpbin.org 80 is using connect mode, the -l one is using listen mode, and -p is --local-port (the port to listen on in listen mode).

Ha, this is more like a puzzle than anything - trying to implement a protocol in a highly restrictive environment.

For anyone looking for an easy way to spin up a command line web server, python has one built in, so if you are running linux usually you can do:

`python3 -m http.server`

From a terminal and it will spin up a web server that serves the current directory. I use this all the time for quick tinkering vs. installing/configuring nginx/node.js/caddy or whatever the cool kids are using these days.

The biggest issue with `http.server` is it's a blocking single-threaded server, so it can serve a single client at a time.

This makes it really really not great when trying to serve pages (with various attending static files), or to ad-hoc share large resources with other people on the network.

A single-threaded server is fine if you're only mildly IO bound, which is the case when you're serving files-as-resources. My understanding anyway is that http.server is explicitly not for production use!

> A single-threaded server is fine if you're only mildly IO bound, which is the case when you're serving files-as-resources.

It's fine if you're also evented, but that's not the case here, http.server literally can only interact with one connection at a time, the next connection(s) will be queued up waiting for http.server to finish its thing and accept them.

> My understanding anyway is that http.server is explicitly not for production use!

Sure, but "I'm at a lan party and have the installer for $game so we don't explode the 'net connection" is not the sort of "production use" that's intended by this statement.

Neither is "I want to open this HTML file I wrote which has a bunch of static assets booting up the webapp".

It also means a mis-configured client that doesn't close the connection can hang the server.

Looking at you HttpClient-on-esp8266...

We had an engineer add it to his unit tests and started blaming the network team when everything started slowing down. Took me two days to convince him to try another server.

>We had an engineer add [python http.server] to his unit tests

I think I see your problem.

He was actually quite good in many respects, just had a massive blind spot here.

Parent does point out they use this for quick tinkering.

Yes, and I point out issues I've hit when using this for quick tinkering.

If you have access to a busybox executable, it comes with a quite capable little httpd with /cgi-bin/ for CGI apps. If you have a wifi router, the web interface might be running off this :) (https://git.busybox.net/busybox/tree/networking/httpd.c)

I use darkhttpd from time to time, it supports streaming: https://github.com/emikulic/darkhttpd

and here's lots of alternatives since that solution has issues

https://stackoverflow.com/questions/12905426/what-is-a-faste...

Also, the python solution is only relevant if python is already installed. Something that's not true on Windows and is or will soon be true on MacOS.

> Something that's not true on Windows.

Getting python up an running on Windows is fairly trivial though. You don't even need local admin permissions if your machine is locked down by your IT dept.

Will Apple ditch Python in the future? Typing on an M1 Air and I`m pretty sure Python came per-installed.

macOS 12 dropped PHP. It still has /usr/bin/python (CPython 2.7), but - any app the uses it triggers a message box about the the developer needing to update their app - before any REPL session it prints

""" WARNING: Python 2.7 is not recommended. This version is included in macOS for compatibility with legacy software. Future versions of macOS will not include Python 2.7. Instead, it is recommended that you transition to using 'python3' from within Terminal. """

There is no version of Python 3 included with the OS. The XCode app/developer tools include Python 3.8. I expect macOS 13 won't include any Python interpreter out the box

Why does it matter if it's included, it's trivial to install and extremely broadly useful?

I don't consider python trivial to install compared to other solutions. Python is much larger than many of the solutions listed in that S.O. thread.

Another option: https://github.com/blueness/sthttpd

But, bash can't do listen sockets.

"loadable accept builtin"

Ahh.

Gawk can be a web server also, typically just "as-shipped", without any new extensions. Doesn't scale well though :)

https://news.ycombinator.com/item?id=22085459

This was very interesting to me - I was expecting using linux socket virtual FS paths to create the socket.

Pretty printed: https://gist.github.com/tyingq/4e568425e2e68e6390f3105e58878...

The /inet/tcp/8080/0/0 is a gawk-ism, versus a Linux thing:

https://www.gnu.org/software/gawk/manual/html_node/TCP_002fI...

Unfortunately, gawk doesn't let you control the listen/accept part of it, so it doesn't scale well. Not sure why they did it that way. A separate accept() call would have made it actually usable in a non-toy way.

Probably designed that way on purpose (or at least post hoc) so that a company wouldn't hit a wall with 10,000 users on the gawkttpd.

This is how you get stuff like hiPHoP.

I appreciate the ingenuity here given the other recent posts, but this still requires executing an external application to perform the accept() part.

Definitely closest to the mark though, kudos!

Basically yes, but it's a loadable builtin.

This script taught me a few things! Since it's kind of impossible to google, what does this construct mean?

   while :; do

It's clearly a while true loop, but why does an ascii crying face produce true?

: is a sort of 'null' function in bash and bourne shells. It also always returns 0 so it's an alternative to 'true'.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...

See https://stackoverflow.com/questions/3224878/what-is-the-purp...

    $ type :
    : is a shell builtin

Bash also has the equivalent "true" as a builtin, and Coreutils provides one as well, allowing to write this instead:

  while true; do

true is also a POSIX utility (not builtin), so it can be used in POSIX-y sh scripts as well.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/t...

The other common idiom in scripts that uses the : is

  : > $LOGFILE

which creates a zero-length log file, and removes the contents if it already existed.

https://stackoverflow.com/a/3224910

You can search for this.

https://duckduckgo.com/?q="while+%3A%3B+do"+bash

You get decent results if you put it on quotes.

What's impossible to google about that? "bash colon" gives tons of results

Including the somewhat unexpected "Bottoms Up Bash - Colon Cancer Prevention Project".

I used to see value in scripts like this, but often these Bash scripts do quite the opposite. It may be readable for those that only know Bash, but you'll probably be happier if you learn Go or C. There are also a good amount of lightweight C web servers like darkhttpd.

It's written in C: http://git.savannah.gnu.org/cgit/bash.git/tree/examples/load...

This is like saying my Python app is written in C because Python is C.

Definitely scary, but I love things like this. Takes a lot of creativity!

There was a post a bit ago that had a one liner for a basic web server using net cat:

  while : ; do cat page.html | nc -l 80; done

or even

    while :; do nc -l 80 < page.html; done

what is the difference in these two approaches?

I always use the pipe, it is intuitive to me to think left-to-right, output of this becomes input of next. But I always see stackoverflow answers using the other approach.

The difference is in the first code example, you're launching both cat and nc.

In the second block, you're just launching one program that reads input, not two. It's slightly more efficient.

I also tend to build large *NIX pipelines using `cat $foo | [...]`, but it's some times regarded as bad form. See: "useless use of cat" for more examples.

Aren't you just using another built-in to read the file contents then?

Maybe it's lighter than cat, but whatever code path is triggered by the `<` can't be zero-cost

If you want to do left-to-right, you can do that:

    while :; do <page.html nc -l 80 ; done

Yeah sure. I have see it, but basically this one use bash builtins instead of external commands.

I had this same idea, but just haven't gotten around to implementing it - so thank you for this reference! This also gives me the perfect excuse for finally getting around to implement automatic dynamic loading of Bash builtins for my Bash package manager

This would be very useful as a fallback - in case something like `socat` isn't installed, this can be used to display some sort of dumbed down version of the site

I follow some projects from you on github. Keep up the work!

Falling back to something like socat, would destroy the whole idea behind it. I think the best way is patching accept and shipping it with the project. Like this no need to fallback.

And considering using something like socat, will force you to fork each request into a new process, which will be slow as hell. I worked before on a web framework, which was used behind httpd or nginx as cgi, it was fast. But the moment where you get more connections it was slow as hell.

The idea of this project is writing some kind of fastCGI (like php-fpm), but for this i would need, to allow multithreading etc..

The other day I wrote this bash code after getting frustrated with the password generators available to me, just so I had something to use right in the shell:

https://gist.github.com/pmarreck/f4dbb02396c4532762a7a64511c...

It actually took an entire afternoon and went through a couple iterations, and I hadn't even written a test for it yet (and it just so happens that I started an MVP shell-script testing library for just such an occasion when I ALSO got frustrated with the bash-accessible testing libraries available to me, also in Bash: https://github.com/pmarreck/tinytestlib)

But then I immediately felt bad. I could have written all of this in less than an hour in a language like Elixir, with test coverage (and the testing/assertion lib is built-in, so it would have been free).

It is impressive and code-golfish that someone can "write a webserver in pure Bash" but Bash is a crap language to script in at this point, and adding more Bash code to the world just seems wrong now. Bourne shells were invented in the 70's (!), long before many language advancements and understandings, and they thus have MANY warts (many of which we're familiar with).

The next time I feel the urge to code anything more than a one-liner in Bash, I'm writing it in elixir script https://thinkingelixir.com/2019-04-running-an-elixir-file-as... and just sucking up the extra run time (or compiling it into an executable with something like https://github.com/spawnfest/bakeware/ and sucking up the extra megabytes), and then MAYBE writing a wrapper function for my .profile

Incidentally, if you DO want a shell that "tastes like" Bourne but is functional and thus sucks a lot less, check out https://github.com/wryun/es-shell, or my fork of it https://github.com/pmarreck/es-shell/

Bash is not mean for kind of projects like this. I even develop on other languages. But i like bash. A good bash script, is like reading a novel, because it's challenging to have a clean script.

This is my favorite "reference" bash script: https://git.einval.com/cgi-bin/gitweb.cgi?p=abcde.git;a=blob...

...it feels like "if it's possible to do in bash, this does it".

5500 lines of bash. Wow.

I get it. The challenging part is fun. You can learn all the corner cases, the clever bash-isms, and then show off. But the code you end up with is essentially unmaintainable, especially since you don't have any test coverage, so...

Well that's true. I took me two hours to get this done. Since accept was not clear at the beginning. But it will be cleaner in the future, i'm still not convinced how it works currently.

You could be a doctor! "I don't really know why or how this works, but I'll prescribe enough for you to last a month".

Here is a fun one I made:

https://github.com/russellballestrini/bash-kira/blob/master/...

Kira (bash-kira): a small bash script for killing programs which run too long.

> and it just so happens that I started an MVP shell-script testing library for just such an occasion when I ALSO got frustrated with the bash-accessible testing libraries available to me

I found Bats straightforward and simple to use, never felt the need to reinvent the wheel there.

BATS is

1) bloated

2) doesn't use a very nice syntax (hint: bashisms are not a very nice syntax). Compare BATS assertions with this for example (the test of my test library, which uses the library's own functions): https://github.com/pmarreck/tinytestlib/blob/yolo/test Call me crazy but I think `assert_equal "4" "$result"` is a lot more readable than `[ "$result" -eq 4 ]`

3) doesn't let you assert on all of (return code, stdout, stderr) from running a single command. I created a `capture` function to do that and bring those all into named variables that you can then assert on.

But overall I wanted something tiny that you could easily include alongside a bash script instead of establishing as a new dependency you'd have to install separately. Honestly, I feel that functions like assert_equal, assert_not_equal, assert_success, assert_failure, assert_match and assert_no_match (which all raise on failure) should be built into every programming language's standard library.

I know that you can sort of get by just using `set -e` and just do a comparison operation, but...

Sort of related, if you do "man read" you the the huge "bash" man page. It would be nice if man could be improved to take you to the "read" command within that page. Or a new command "manbash read"

Well just type: help BUILTIN

Example : help read

Thanks I didn't know! So many years of searching for "read" (etc) in the huge bash man page and, of course, in the case of "read" finding many false positives. Still it would be nice if `man` could be improved to know about builtin commands - the obvious thing is something like an html anchor in the big bash man page for each built in.

This prints abridged usage help, though. You can use `info bash read`, but confusingly it might not always contain the same content as manpages.

I like those unorthodox, against-cargo-cult approaches. Thx!

     printf '%s\n' "$(<$tmpFile)"

Why not just

     cat $tmpFile
     printf '\n'

to avoid external calls

If Bash is Turing-complete could we just make a Python to Bash transpiler?

The project has been patched, now we can handle multiple connections!

Right, time to make this into a Docker image and ship it to production

I wonder if it's faster than the same thing in javascript...

Most of the node.js solutions I'm aware of would support http/1.1 and async so I can't imagine this faring well in a performance comparison.

Well benchmarking it os currently not possible. I will try it in a few days and provide some benchmarks.

The famous HN lack of humour shows its head :)

one day this code will end up on a small/home office router

…and I like to live dangerously.

Reply on news.ycombinator.com