Go vs. Python for a simple web server | Hacker News Reader

Go vs. Python for a simple web server

blog.kowalczyk.info

51 pointsfogus20 comments

While this article discusses some pros/cons of Go for implementing a web server, I'm surprised there's absolutely no mention of how fast his working (?) server runs or the memory footprint relative to python.

When I read "In addition, threading in Python performs poorly due to GIL." and then just leaving it at that, I wasn't expecting any sort of performance or resource usage investigation.

When I read "I would seriously consider using Go for writing server side of web applications as it provides a right balance between speed and memory efficiency of the running code", I was expecting at least some number to back it up.

Agreed. If we want to nitpick further regarding the GIL - Python boosters could state that the GIL is just a implementation detail of cPython that is not a problem in other implementations, like Jython or IronPython.

"don’t suffer Python’s terrible per-object overhead"

Yet another example of where I'd love to see some numbers.

    kragen@inexorable:~$ python
    Python 2.6.6 (r266:84292, Sep 15 2010, 15:52:39) 
    [GCC 4.4.5] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os, commands
    >>> print commands.getoutput('ps u %s' % os.getpid())
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    kragen   20823  0.3  0.1   9636  3780 pts/4    S+   14:00   0:00 python
    >>> x = [{} for ii in range(1000*1000)]
    >>> print commands.getoutput('ps u %s' % os.getpid())
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    kragen   20823 10.8  7.7 168176 159756 pts/4   S+   14:00   0:02 python
    >>> (159756-3780)*1024/(1000.0*1000)
    159.719424

So the baseline is about 160 bytes per dict. Objects, by default, are built on dicts. Objects with __slots__ can be a little more efficient.

You can look at the memory usage here: http://shootout.alioth.debian.org/u64/benchmark.php?test=all...

It's a fair point. However, when I did that port, almost a year ago, I did it to understand how Go compares to Python in terms of writing the code. The result was fast enough, I didn't bother benchmarking.

Plus, whatever the results were then, Go's compiler and runtime has improved in the past year, so the data would no longer be indicative of current performance. And the compiler is still being improved.

"Overall, the experience was positive and I would seriously consider using Go for writing server side of web applications as it provides a right balance between speed and memory efficiency of the running code (much closer to C than to e.g. Python) and speed of development (much closer to Python than to C)."

It seems these days that there is a confusion between the web server (often a reverse-proxy), the web application and the thing in the middle (gateway/pipe/container/app server). While I would certainly consider Go for the implementation of a reverse-proxy or an app server, it isn't clear that Go is a good choice for the web application itself, especially when you consider that most of the time is spent in the DB and in the cache...

Anway, the article mostly talked about Go and not about Python...

That confusion is because there are a lot more options now, what with there being embedded (performant) HTTP libraries so you can run your stack without a separate server, or you can embed user code into a traditional server, or many other combinations. Async stacks do not a bit in the middle. And some NoSQL databases like Couch can embed the web server and middle parts of the code.

Go has an interesting combination of lightweight threading and a high quality performant HTTP server, so potentially it is good for implementing the http serving and application layers.

"Speed and memory efficiency: I haven’t measured" - ok. "In addition, threading in Python performs poorly due to GIL" - ok.

Last time I created a simple web service with Go it leaked like a hell.

If you are using the 32bit compilers, there were issues with the garbage collector for a while, in any case you should always use the 6g 64bit compilers for any serious work, they are much better tested and generate more efficient output.

Other than the issue with the 32bit compilers, I don't remember hearing of any leaks in Go.

"efficient Python web servers (like Tornado) use event driven way of handling HTTP requests. The downside of that is that there can only be one HTTP handler function executed at a time. If that function takes a lot of time, it blocks all other requests."

This statement needs clarity. If the handler is CPU-intensive, the design of the server (threaded vs. AIO) is irrelevant. Both will perform equally, the only difference being that on an SMP system, the OS could schedule the thread on a different processor. This is why the typical deployment model for AIO servers is to start one server per processor and route incoming requests to a proxy to balance the requests among the available servers.

On the other hand, if the handler is dependent on slow network services (e.g. database or downstream API service), the AIO-based server will easily be able to handle additional incoming requests while waiting for the other tasks to complete, subject to memory. This is the power of non-blocking I/O function calls.

the only difference being that on an SMP system, the OS could schedule the thread on a different processor.

There's a bit more to it than that. The OS will frequently block and preempt the long-running thread even on a single CPU system, allowing other, shorter requests to be processed in the mean time.

I'm assuming for this example that every request has equal CPU impact. Preempting the running thread in this case won't improve overall request throughput on a single-CPU system. Adding concurrency here will simply increase response latency.

Benchmarks? Code? Anything remotely empirical?

If you're looking for something useful, I am afraid that you are looking in the wrong place. Similarly to the last post by the same author, this is just an anecdote.

Should say that it compared Go vs. Tornado. His only real problem with Python was concurrency which could be handled much better with Eventlet or Gevent. This will also drastically cut down the line count of the source.

Great! Someone hacking something useful in Go. We need more non-academic, practical code written in Go. So, Go Krzysztof, Go!

Most Go code I have seen was non-academic and practical. See for example doozer: https://github.com/ha/doozerd and camilstore: http://camlistore.org/

Reply on news.ycombinator.com