Go as an alternative to Node.js for Very Fast Servers | Hacker News Reader

Go as an alternative to Node.js for Very Fast Servers

techblog.safaribooksonline.com

189 pointsmeirish139 comments

I played around with a go server to do some simple scaling numbers - looking at possibly using go to implement a large-number-of-idle-connections notification server.

I found the (good) result that I could spawn a new goroutine for each incoming connection with minimal (~4k) overhead. This is pretty much what you'd expect since a goro just needs a page for it's stack if it's doing no real work. I had something like 4 VMs each making ~30k conns (from one process) to the central go server with something like 120k conns.

I found one worrying oddity however. Resource usage would spike up on the server when I shut down my client connections (e.g. ctrl-C of a client proc with ~30k conns).

Reasoning about things a bit, I think this is due to the go runtime allocating an OS thread for each goro as it goes through the socket close() blocking call. I think it has to do this to maintain concurrency. So I end up with hundreds of OS threads (each only lives long enough to close(), but I'm doing a lot at the same time).

Can anyone comment:

- is this guess as to the problem likely to be correct?

- is this "thundering herd" a problem in practice?

- are there ways to avoid this? (Other than not using a goro-per-connection, which I think it the only idiomatic way to do it?)

My situation was artificial, but I could well imagine a case that losing, say a reverse proxy, could cause a large number of connections to suddenly want to close() and it would be a shame if that overwhelmed the server.

> I think this is due to the go runtime allocating an OS thread for each goro as it goes through the socket close() blocking call. I think it has to do this to maintain concurrency

I highly doubt that it is creating a thread per goro on client disconnect. If you have a minimalish example of this, the golang mailing list would be very interested in working with you to identify what went wrong and create a patch if it is an issue with the Go implementation.

In case you didn't see it, I commented above with a link to a bug.

Blocking system calls spawn OS threads in Go, which can be cached and recycled for new goroutines. You don't see this if you code to pkg/net because it multiplexes i/o with a select/kqueue goroutine, but you'll see it right away if you code directly to the syscalls.

Close isn't a blocking call, though.

Close is annotated as a blocking syscall in http://golang.org/src/pkg/syscall/syscall_linux.go#L810

Is there a guarantee on Linux that close can't block? (I suppose it depends on the file type and the definition of blocking.)

Only if SO_LINGER is set.

It seems so.

https://code.google.com/p/go/issues/detail?id=4056

An interesting point raised there is that if they instead used a limited thread pool for all goroutines to share when making OS calls you could produce deadlocks.

I'm curious what calls could induce deadlock. I figured everything used a nonblocking API internally and the runtime conferred blocking semantics.

As I understand it, the runtime has to clone a new thread for each sync call into the OS.

That is the (only) mechanism by which a goro can perform such a sync operation without risking blocking the process.

It's entirely possible under some workloads that those blocking OS calls require other goros to make progress (think pipes), so this could result in deadlock (although for some workloads it wouldn't I guess).

I'm groping blindly here, but from the references in that message to cgo I think it might have to do with calling into foreign code. Presumably Go's scheduler can't yield when you're inside C code (I'm not aware of any lightweight thread system that can achieve this) so you have to block there, which means you can no longer schedule other goroutines on that thread, and thus you have to spawn a new thread if you want to get any work done. But I'm just guessing here.

While JavaScript drags the scars of its hasty standardization around with it, Go was designed very thoughtfully from the beginning, and as a result I find that it’s a pleasure to write.

This is very true. Go is a pleasure to write. In fact, it's such a pleasure then when you hit something that wasn't really well designed it's horrid.

Can I ask what language you are used to? I hear how nice go is as a language a lot, but coming from haskell, go is hideous in comparison.

Go is going to look hideous to you because you're probably expecting functional things like list comprehensions (which Go doesn't have) and a very intricate type system allowing for things like generics (which Go doesn't have either).

Go code will look a little less DRY to you as a result, which is a fair criticism, but it makes up for that by being incredibly opinionated (that's a good thing), being incredibly easy to prototype in, and being incredibly easy to refactor painlessly.

>Go is going to look hideous to you because you're probably expecting functional things like list comprehensions

Nah, list comprehensions are just syntactic sugar and not used much in haskell. Python seems to encourage their use a lot, but you hardly even see them in haskell code.

>a very intricate type system allowing for things like generics (which Go doesn't have either).

That is definitely one of the big problems, but I take issue with the characterization of that as needing "a very intricate type system". Parametric polymorphism is very simple, and has been a completely solved issue for a very long time. There is simply no excuse for a brand new language to be decades behind on something so easy to do right.

>being incredibly easy to prototype in, and being incredibly easy to refactor painlessly.

Those are actually two of the other big issues going from go to haskell. Go is harder to prototype in, and it is easy to add bugs when refactoring because the type system is so poor.

> Parametric polymorphism is very simple, and has been a completely solved issue for a very long time. There is simply no excuse for a brand new language to be decades behind on something so easy to do right.

Go has addressed this; their approach is Go's interfaces, which combines the best of all worlds: duck typing with static type checks and type inference.

> Go is harder to prototype in, and it is easy to add bugs when refactoring because the type system is so poor.

This is where we'll have to agree to disagree. It's definitely not harder to prototype in - and I say this as a functional programmer - and if you find the type system to be inadequate when refactoring, it sounds to me like you're trying to write idiomatic Haskell in Go. Go's type system, by design, stays out of the way - if you're writing Go idiomatically, you really shouldn't be thinking very much about the types as you write them.

As for generics, this gets beaten to death on every single Go post on HackerNews. Yes, Go would ideally have generics. Yes, there are tradeoffs involved. Yes, those tradeoffs have been explained by the Go developers at length. Yes, they would be open to including them in the future, if somebody addressed the existing concerns. No, nobody seems to mind that they're missing from the language as-is, given those tradeoffs.

You are contradicting yourself about parametric polymorphism. First you acknowledge it doesn't exist, then you claim a limited workaround solves it.

>It's definitely not harder to prototype in - and I say this as a functional programmer

Have you used haskell to make the comparison?

>if you're writing Go idiomatically, you really shouldn't be thinking very much about the types as you write them.

I don't understand where you are coming from here. I am not thinking about types, that is why I need the compiler to point out when I mess them up. The problem is go has such a limited type system, that you have to change much more code when you refactor, and the type system is inadequate for catching many errors, in particular dealing with error handling. The combination makes go worse for refactoring than haskell. It is certainly much better than python for example, but you seem to be convinced that go is the top of the spectrum and nothing can exist above it.

I have been wanting to learn Haskell for some time. Is Real World Haskell still the best book for the language? Could you point me to some well written libraries/projects that are considered idiomatic haskell? Would appreciate the pointers.

I would definitely start with learn you a haskell. It makes a much better introduction to the language than RWH. RWH is great when you have learned the basics and want to start tackling bigger problems.

Xmonad is a pretty common recommendation for looking at "good haskell code", in particular the overall design and how they keep the IO part minimized and isolated so the bulk of the application is easier to unit test. I think the standard libraries the come with GHC are good examples too.

There's lots of books: Hutton, Hudak, Thompson, Richard Bird. I have a ugly draft/link dump for learning

http://isthishaskell.blogspot.com/2013/02/tips-on-learning.h...

> Go's type system, by design, stays out of the way - if you're writing Go idiomatically, you really shouldn't be thinking very much about the types as you write them.

I'd wager that if you aren't thinking about what your inputs and outputs are going to be at every step, you're introducing bugs or working harder than you have to. A strong static type system like you find in Haskell formalizes that so that it's required, but even in the languages I write most often (Python and C, which get bashed constantly for their type systems), this is just what writing solid code is about.

In Python, I may not be thinking exactly "what is the type of this thing, foo", but I am asking myself "okay, I'm trying to iterate this thing, is it actually iterable? How do I handle when it isn't? or guarantee it to always be an iterable ?".

actually, the thing i find most lacking in go (although granted i've only done a few toy programs in the language) is sum types. it would make error handling a lot more pleasant, for one.

i'm excited about rust, which did go the algebraic datatype route.

I'll tell you that coming from Python/C it looks very nice. I'm actually learning haskell atm (use xmonad for a window manager), I'm afraid that if it lives up to it's hype I won't want to program in anything else.

Also playing with Racket, Clojure, and good ol' Common Lisp.

Define hideous.

"Extremely unpleasant"

Here's a comparable Haskell server, warts and all

    {-# LANGUAGE OverloadedStrings #-}
    import Snap
    import qualified Data.ByteString as BS
    
    main :: IO ()
    main = httpServe (setPort 8000 emptyConfig) $ writeBS $ BS.replicate (1024*1024) 100

I'm benchmarking it currently, but my laptop's network stack seems to break ab. It's also probably faster to build the response incrementally using an Enumerator, but I've never used Snap's Enumerator library and this is slightly closer to the design being tested in the other servers since it'll allocate the whole bytestring instead of writing it lazily.

I tried this, the other Haskell version, and the go and node version from op -- with some rather ridiculous ab-values -- the end result was that both of the haskell versions crashed after around 8k requests, while both the go and node-versions completed -- with no missed requests.

I don't have the full numbers (didn't log them) -- but running "ab -n 100000 -c 1000 http://localhost:8000/ -- nodejs completed in 160 seconds, go in 170 seconds. Nodejs had a few requests around 8 seconds, and go had a worst time of almost 5 seconds. (This is on an old desktop, with a core 2 duo - roughly 6000 "bogomips" pr core, two cores).

I'm guessing the haskell solutions ran out of resources, but I'm not sure.

Not sure, I'm trying to replicate it but I'm having a tough time with OSX's network stack. I can't siege or ab very well. The Haskell server tends to stay under 6Mb on my machine though. I'll keep stressing it.

http://imgur.com/b89bxST

Because this isn't the same benchmark, did you compile with -O2? The Go version allocates a 1MB slice once and sends it to every user. The Haskell version literally states that it should make a bytestring during every request. The compiler might optimize it away with the right flags.

I tried the benchmark as in the article, and the above Haskell compiled with '-O2'. With Go I usually get around 1320 reqs/second, with Snap around 1100. Both using four threads.

With a single-threaded Warp instance, I could get around 1350 reqs/seconds, though it failed dramatically when I used more threads.

No, I didn't (see my other comment). I expect (especially Haskell) to be correct without -O2 -- being slower isn't a problem - crashing - not so much.

Tried it now -- both haskell versions crash with -O2 as well (and ab -n 100000 -c 1000 -- so not the same benchmark as the original -- and a rather silly test).

I studied Haskell for a decade with the goal of actually using it for production software. When I finally found a nice gap to try it, the experience was mostly awful.

Go was such a huge relief after that horrible catastrophe language that seems to still continue to wreck new generations. Please, please, don't poison your career on focusing on a single language, especially one as disturbing as Haskell.

What were the specific problems you had with haskell?

I had several small things (laziness, poor libraries [in 2009], failing to get anyone else in the organisation even the slightest bit interested, my own lacking skills) but the overall worst problem was the feeling of the disconnect between the language and the host running my program. That is of course more or less always an issue when one uses a higher level language, but I felt Haskell was so far off that it disturbed me.

Also, the negative effect one gets after going back to one's own code after several months seemed much much worse in Haskell. Haskell code is too thick, too concise, not unlike mathematics. I find it ironic that the community, apparently, values that.

I bet others have had better experiences, but I definitely did not. Hanging around with Haskell clearly taught me many good things about program design, but I don't foresee myself wanting to ever write anything on the platform ever again.

>Please, please, don't poison your career on focusing on a single language

My career is likely older than you, I think it'll survive the horrors of using a good language. I've been running production web sites on haskell for a year so far, and it has been great.

"My career is likely older than you"

Your career spans over 30 years? If so, I applaud you and of course withhold my rights to criticize your choices without knowing more details.

I've done Java, PHP, Objective Caml (in college for a few years), some basic C, Ruby and Javascript before and I concur. Go is a real breath of fresh air. I have the same feeling I have when I code with Ruby: that sense that the language works WITH me, that the whole experience is smooth and seamless.

I wish Go stays on that path for a long time. And it never gets TOO big, which makes for awful communities.

Maybe you didn't do much OCaml? I've found it to be mostly significantly better than Go, especially with it's type system. Go's type system is relatively limited and ad-hoc where OCaml's is extremely elegant, simple and consistent.

In particular, OCaml has an awesome module system and a great take on structural sub-typing combined with proper type inference and sane parametric polymorphism.

Having actual algebraic data types is night and day to Go's more limited structs. OCaml also has a good object system which people don't use too often but can come on very handy.

I've written a moderate amount of OCaml and some Go.

While I agree that the former is a nice language, it doesn't compare favorably to Go as a development suite: the standard library is terribly designed, and the package management is rather poor compared to Go's (which is about as great as it gets).

Now there is "OCaml Batteries Included" which is supposed to mitigate that, but it appeared after I stopped actively using OCaml so I can't comment on that (incidentally, I dislike many of their choices).

Also Go has many of the features that make programming in ML a pleasure, many languages these days have them, which is a good thing.

Personally my tool of choice for many tasks would be OCaml with Go's package management system, and perhaps a syntax closer to M-expressions (like Mathematica).

I love the package management in go and agree it is pretty good, but the lack of versions make me nervous. I do wish they reacted quicker to common criticisms like this and the missing generics, but I guess they want to take their time and do things right. What there is of go has been a pleasure to explore though.

Would you mind expanding on what is good about m expressions and how that could apply to go, I'm not familiar with them?

See [0] for an introduction to M-expressions, which was initially an alternative syntax for Lisp, but have found its way into APL and Mathematica.

Please note I'm suggesting using that syntax for OCaml, not Go.

In short, I want to write "f[x;y]" for "f x y". On the first sight it appears this adds a bunch of noise, but there are many advantages:

1. You now have much less parens resulting from nested function calls;

2. You get partial application by any argument, not just the last. map[;list] is a "functor" that applies its argument to "list";

3. Corollary to this, chaining functions together is now much easier even if you need to supply a parameter other than the last: f[;x]$g[y] (assuming $ stands for apply; it might be wortwhile to take empty space for application)

4. Further you are able to unify many aspects of the syntax (if[cond;true;false]), which makes parsing easier for both humans and computers.

[0] http://en.wikipedia.org/wiki/M-expression

OCaml should definitely move toward adopting either Batteries or Jane Street Core as its standard library; they offer a lot more functionality.

For package management, I haven't had the chance to really try it yet, but I'm hearing good things about OPAM.

Finally, Rust just looks awesome, I'm hoping that the language stabilizes before the end of the year, as I'd love to do my master thesis with it.

As great as OPAM might be (neither have I tried it), it has one fundamental downside compared to Go: it is a package manager for OCaml.

Package management is one of those things that are better off when there is just one.

I'm pretty sure OPAM will become the de facto standard of the OCaml world; most people don't want to deal with the other ones.

If you like OCaml but find Go lacking, you should check out Rust. I'm more of a Haskeller than a OCamlist, but the Rust compiler was originally written in OCaml, and it's influenced the language a lot.

I find this recommendation interesting because I am developing an OpenPGP keyserver in Go, called Hockeypuck (https://launchpad.net/hockeypuck). I started the project because of problems I ran into with SKS in a very specialized, write-heavy use case. SKS is developed in OCaml.

I've been porting the reconciliation algorithm in SKS to Golang so Hockeypuck can peer with SKS servers. I think it will be very useful in other applications beyond keyservers. The mathematical definitions of finite fields and polynomials are elegant in OCaml, and I definitely understand the choice of language from that point-of-view.

If you want to see a comparison of finite-field arithmetic and polynomial factoring in Go vs OCaml, my recon port is in a separate project called conflux (https://github.com/cmars/conflux). It's an incomplete work in progress, needs tightening up, tail-call elimination, etc. but early feedback is welcome.

As a developer unfamiliar with OCaml & without much formal mathematics background, I had a hard time understanding some of the intent of the SKS sources -- in those cases, conflux is a straight-up port from OCaml with unit-tests also ported from SKS to back it up. Some of it, I understood the math concept, but not the OCaml, so I ported from SymPy instead.

Wolfram Alpha was also helpful to validate my work and create test cases -- it does polynomial factoring over finite fields!

Yeah, like I said, it was mostly during 2 years in college (in France in the close-knit network of research institutes and universities where the language was actually born) and OCaml is a wonderful language. It has numerous upsides (type inference, the mixed functional/OO paradigms, non-verbose, relatively fast, the awesomeness of pattern matching).

I was mentioning it in my previous comment to provide perspective on my opinion of Go.

I am a big supporter of the idea that all languages have their advantages anyway :)

> The biggest promise that Node makes is the ability to handle many many concurrent requests. How it does so relies entirely on a community contract: every action must be non-blocking. All code must use callbacks for any I/O handling, and one stinker can make the whole thing fall apart. Try as I might, I just can’t view a best practice as a feature.

Nonblocking I/O isn't just a "best practice" in the sense that consistent indentation is a "best practice," it's a core tenet of the Node ecosystem. Sure, you could write a Haskell library by putting everything in mutable-state monad blocks, and porting over your procedural code line-for-line. It's allowed by the language, just like blocking is allowed by Node. But the whole point of Haskell is to optimize the function-composition use case.

The Node community has the benefit of designing all its libraries from scratch with this tenet in mind, so in practice you never/rarely need to look for "stinkers" unless they're documented to be blocking. And unless they're using badly-written blocking native code, you can just grep for `Sync` to see any blocking calls.

Node: Everyone knows JavaScript, there's a massive community, there are tons of libraries, and you get very good performance

Go: No one knows this language, there's a small-but-growing community, there are enough libraries to get a lot done, and you get even better performance

Java: They are paying me (money!) to write in this language

Yes, but at what point does go transition from obscurity to PG's "python paradox"?

http://www.paulgraham.com/pypar.html

It sure seems Scala's in this "python paradox" land now. My guess is that you need some startups make it big using Go to evangelize it. Google using it is interesting, but I'm not sure it makes it "cool".

Though, I'm not sure Java was ever a language you could use as a skillset filter. Hm.

Pretty much anything Google does is cool. Probably same could be said about Amazon, Facebook et al.

Depends a lot on your interpretation of "Cool" though.

Right.

I get more of the sense that Google is more "impressive" than "cool".

If I were to use "Google's got it in production" I might be told something like "well they can pull this off" as if they have some unattainable smarts or something.

Whereas a "cool" thing is more about just simply taking a risk.

Come to think of it, that's kind of a funny definition. Oh well.

I think a lot of the "coolness" factor of Go comes not from its parent company, but from some of its core developers, namely Rob Pike & Ken Thompson. That gives Go a serious Bell Labs/Unix/Plan 9 pedigree.

I don't follow the mailing list anymore, so I don't know if it already led to the same "cargo cult" fanboyship that Plan 9 sometimes evokes, where a lot of the idiosyncratic opinions of its developers (e.g. "shared libraries are bad") are basically never questioned and repeated almost like holy scripture.

Yes!

I tend to think of "coolness" as a kind of reason behind early adoption, more like "this thing as a large opportunity for catching on".

I'm not 100% sure would make the argument that "Rob Pike and Ken Thompson made it". Instead, I'd say "it's in use at Google, and all of these other startups..." If 1 or 2 of those startups hit it big (e.g. Twitter or LinkedIn kind of big) that might have a bit more of an "Ooo" factor.

Though I have said, "it's incredibly well thought out, look, Rob Pike and Ken Thompson know what they're doing". I don't sense this quite had the gravitas I was looking for yet. Hm.

The mainstream always lags behind significantly in every aspect of life. If everyone's using it, then you have little competitive advantage using the technology.

Go would be one of the first things I'd reach for if there's any chance server-side concurrency would be involved. The language is minimalistic and unsurprising to the extreme. A joy to program in and use.

The Go community on Reddit is already bigger than the Node community: http://www.reddit.com/r/golang (3730) http://www.reddit.com/r/node (3581)

And the Google+ Go community is growing fast and is already larger than the sub-reddit. https://plus.google.com/communities/114112804251407510571

You do see a correlation right?

I think he was referring to the whole JavaScript community.

For comparison: http://www.reddit.com/r/javascript (27,041)

I wouldn't consider subreddits a good way to gauge a tech community's size. Try IRC rooms and mailing lists.

More like: Anything web HAS to be Javascript (cause the calendar says 2013 but apparently it's 1970), no choice so oh well, we'll try Javascript on the server cause God knows using the same language everywhere is a good thing :/

Go: The language is in its infancy, growing at a slow pace for now, bears some promises that are yet to be confirmed.

Java: For some reason people still hate the language even though it's the closest to being the most versatile language around (in every single aspect that makes a good language it ranks well against the others)

> For some reason people still hate the language even though it's the closest to being the most versatile language around (in every single aspect that makes a good language it ranks well against the others)

In every respect that a PHB may care about, perhaps. I think you should examine that "for some reason" more carefully before declaring that all reasons favor Java. Clearly there is something going on there, unless you think everyone who dislikes Java just suffered head trauma or something.

Well, I know lots of programmers who do like Java, you just don't see them that much on Hacker News. Reasons:

- Availability of good IDEs.

- Good dependency management and build infrastructure via Maven.

- Quick and easy deployment via servlet containers.

I am not a big Java fan, but having written quite much code, feature-wise there are not that many advantages of Go over Java. Package management in Go is nice for an early system, but will become a mess eventually, since there is no version management at all. Goroutines and Gochannels are nice for concurrency, but not all that great for parallelization. Java has generics, checked exceptions, and a good garbage collector via the JVM.

I don't hold much hope for the development of Java the language, but the JVM is a great platform, with many interesting languages (Scala, Kotlin, Clojure), that attempt to solve problems that Go doesn't solve.

You and VeejayRampay seem to have misunderstood me. I'm not saying the language is all bad, or that the technology is crap, or that there are not programmers who love everything about it. I am saying that there are reasons that many people dislike Java; it isn't just some sort of blind prejudice.

@VeejayRampay

Considering the love Clojure gets on HN, I reject the notion that Java gets a bad rap due to historic shortcomings of the JVM or ecosystem. No, the language itself is disliked, not the tech. Its constructs and its idiomatic usage. Things that your standard PHB will find difficult to quantify. This is in stark contrast with how the JVM is perceived (from my perspective, it seems to be widely adored).

I'm saying this as someone who currently makes their living programming in Java.

Let me clarify then:

I think that a consensus was built in the mid-90s that Java was a pain in the ass and not an improvement over what was available in that particular field at the time (not as easy as other higher-level languages and not as good or fast as C or C++).

I have a feeling that people over time consolidated that consensus through some form of confirmation bias even though the Java/JVM ecosystem had made strides towards a brighter future. And I say that as someone that doesn't even like Java in the first place (or at least doesn't see himself enjoying coding with it).

But it's hard in 2013 to look at what Java and the JVM brought to the table with disdain. Java is nowadays a very fast language that deals with memory management extremely well (not a good feature in some fields of course, embedded or real-time systems come to mind), that offers a rather sane Java-flavoured OOP paradigm, tons of tools (from IDEs to debuggers to servers) and an extremely well-crafted documentation.

The JVM also enabled Scala, Clojure, JRuby, Rhino, Vert.x, etc.

Now I totally understand and agree that the picture of Java nowadays is still far from perfect but I was more pointing at the slight disconnect between the actual capacities of the language and its perception by the programming world at large. Overall Java and the JVM are pretty damn good tools to use and meet the needs of lots of different niches.

It's not the JVM or the ecosystem, but the language itself. It is clunky and generally irritating to write and read. But the libraries and its niche sophistication (basically enterprise middleware and webapps) are second to none, so people put up with it.

Note: I've been writing it professionally since 1999, currently working a lot with ServiceMix so I'm knee-deep in the enterprise stuff.

Exactly, the JVM and the ecosystem are just fine. There is no particular hate for them, just look at Clojure.

I've been using Go a lot lately. It's difficult to overstate just how much simpler it makes writing highly-concurrent server-type programs. Entire classes of bugs, issues, and puzzles just vanish.

I've been using Go a lot recently as well, and it's rapidly become my go-to language (no pun intended) for a lot of problems, even when concurrency is not involved.

The biggest thing Go gives me is that it's really easy to manage code bases that grow organically - refactoring a project that grows from 50 LOC to 5000 LOC is almost painless in Go - no other language that I've seen has dealt with this aspect of code development so well.

What about Go makes it easy to manage and refactor large codebases? I don't do much Java, but whenever I've watched someone use Eclipse for refactoring, I question why I'm still using vim, because it is just magic and does everything for you.

I should probably write a blog post about this, because it's a combination of a number of things. Primarily, the compiler is incredibly strict and opinionated, so it's impossible to make certain small errors like assigning to lvalues that are never used, importing packages that are never used, etc.

Secondarily, gofmt makes code very standardized and easy to skim. It takes Python's 'only one (obvious) way to do it' one or two steps further, by forcing everyone to to write their code the same way. This makes refactoring a lot easier because you don't need to read as much code each time in order to understand what's going on (or at least, you can mentally parse it much faster).

Finally, the context-free grammar combined with a strong, static type system means that migrating code from an old (incompatible) version of Go to the most recent one can be done painlessly with the 'go fix' tool. This isn't your py2to3 tool - this accepts valid (old) Go code as input and reliably produces valid (up-to-date) Go code as the output.

That last bit isn't actually used in refactoring manually, but I make note of it because very few languages give you anything close to this level of reliability with code modification, which speaks volumes about the design of the language's grammar.

But go ecosystem still doesn't have anything even close to refactoring abilities of Java/C# IDEs. Say, move Java class which is in use in 100s places from one package to another is just few mouse clicks in Eclipse and a lot of pain in case of Go.

Not really. Go comes with code rewriting tools, unsurprisingly since they were used to update the standard library and 3rd party code when the language changed before Go 1.

I don't want to use a language that doesn't come with a parser in the standard library anymore.

So, is it possible to move/rename class/method/field smoothly using those tools? Extract code block to new method?

the gofmt tool has a replacement feature:

`gofmt -r 'InitialName -> FinalName'`

Documentation looks quite incomplete(http://golang.org/cmd/gofmt/). Will it change all imports and usages of class/method/field? Will it resolve all signature polymorphic calls?

"The biggest thing Go gives me is that it's really easy to manage code bases that grow organically"

Very interesting to hear. Any chance you could expand on this a bit more?

Unlike C++, Go enforces that its dependency graph be a DAG. This drastically reduces compilation time, but it also reduces the headache that comes with decoupling highly coupled code, because it restricts the extent to which your code can be messy to begin with.

The package system also enforces (at compile-time) that every imported package be used (and also that every named identifier be defined, which most dynamic/interpreted languages can't do). This applies not just to imported packages, mind you, but to any lvalue - if you declare/assign to an lvalue that's never used as an rvalue later, you can't compile the program.

That's really helpful when refactoring, because it makes it easy just to move a bunch of code between files, then follow the breadcrumb trail of compiler errors (not warnings!) to figure out what still needs to be fixed. The compiler won't tell you everything you need to do, but it's sort of like having a Roomba helping pick up after you while you clean your house manually.

Yes, for those genius programmers who never makes any mistakes, this may not be much of an improvement. But for those of us who don't trust our human brains as much and want to be absolutely sure that these silly errors don't slip through, it takes a huge load off the mind.

Unlike C++, Go enforces that its dependency graph be a DAG.

In C++ header dependencies are also a DAG, since include guards prevent cycles (and multiple includes). What makes Go faster are a few things: (1) C++ headers contain templates, which are slot to compile; (2) Go only looks at direct imports and uses the compiled form of those imports, rather than recursing over their imports (again); (3) Go is simpler to parse; and (4) there is no overloading, so symbol/method resolving is simpler.

I also think that the advantage is often overstated. C++ is a nightmare in this respect, C programs and libraries often compile very fast (on my current machines, running configure often takes much more time than the actual compilation), the same applies to e.g. Java code.

That's really helpful when refactoring,

And annoying for testing, the printf example has been beaten to death. (Yes, I know that you can add a line such as var use = fmt.Println).

But for those of us who don't trust our human brains as much and want to be absolutely sure that these silly errors don't slip through

It's always surprising how Go fans can sell a feature that any strong statically typed language always had (easy refactoring by letting the type system work) can sell as something unique and new ;).

Compilation units can rely on each other through forward declarations, so the dependency graph is not acyclic. Header exclusion does form a DAG within a compilation unit, I guess.

A forward declaration is not really a compile time dependency.

If you are forward declaring a function, you are just promising that it is present during linking. So, during compilation it is not an edge in the graph.

If you are forward declaring a data type such as a class, a full definition needs to be visible at its first use or you are using the type as a pointer:

    class A;

    class B {
      A *d_a;
      [...]
    };

In this case it is not really a dependency either, since the compiler does not need to know the size of A, since d_a is a pointer. When you start to dereference d_a, its definition needs to be fully visible, which is done via headers, which are a DAG through guards.

Could you give an example where C++ dependencies are not a DAG during compilation?

Yes, but it is still a dependency. By which I mean the software won't run if you don't supply the necessary thing at resolution time (which is possibly quite late: well into runtime, if you are on a system with lazy linking). Go (and some other languages, like OCaml) enforces that such dependencies form a DAG: C++ does not.

I don't think this has much to do with compilation speed though.

I have exactly the same experience. Very pleasant to write programs that are going to be used by more than 10k users at once.

> There’s no arguing about whether to use semicolons, or putting your commas at the front of the line — the language knows what it wants. It’s a built-in hipster suppression mechanism.

Major point for saving man hours right there.

The fact that Node.js is being used in this equation says a lot about how much impact and penetration it has achieved in a rather short while.

Personally I hope that Go does just as well, if not a lot better. I am a bit of a fan of both.

I think it says more about the bias of the writer, he seems to assume that Node would be the default choice and that something like erlang/scala/clojure/go would be alternatives. That may be true for someones sideline project.

If he's a HN reader, he's probably assumed that Node is the default choice these days, based on the articles that come up.

Not really no. It only says a lot about how much advertising has been done recently about the webscaling capabilities of Node (like it's anything special). I think people are starting to realize that as good as it is, it simply isn't the only way to achieve results like these (see Vert.x, Tornado and tons of other projects with comparable capabilities).

Clojure is another nice alternative for fast servers, and using a concurrent, immutable and functional language is a huge win. http-kit is a good example of such server: http://http-kit.org/

Here's a haskell comparison (hint: it does very well).

https://gist.github.com/jamwt/5017172

Haskell was ghc 7.6.1 with ghc --make -O2

Go is go1.0.2 with "go build".

Warp is very nice.

As noted above - with ridiculously large values for ab this one crashes (although I didn't compile with O-parameters). I think this (and the other haskell solution) ran out of resources.

Both the go and nodejs versions completed without problems.

I was a little disappointed -- I was actually hoping I'd see comparable performance -- even if it is a silly test.

I think it is interesting that simple, idomatic code in go and nodejs didn't crash -- not sure what assumptions might be "wrong" in the underlying haskell code (I'm guessing if anything should be "fixed" it is in the web server libraries used).

I have never understood the focus on speed as a selling point for Node. It may well be very fast, but it seems to me that the primary selling points would be the ability to share code between client and server and that you can start coding server side without learning a new language if all you know is JavaScript.

Compared to Python and Ruby, node.js is quite fast by the simple virtue of having a JIT (in the most common implementation anyway; of course there are JITs for Python and Ruby but they aren't the mainline implementation).

Which is a shame, since PyPy is really an excellent project.

I never thought of it as "faster-than-thou" but rather "faster than you expect" and "fast enough for real work."

First off, while some sharing between client and server happens, that tends to be an edge case in my experience. The roles of client and server, and APIs available to each, are rather different. I.e. the environment of the browser and node.js server aren't homogenous.

Second, "start coding in XXX without learning a new language" is a terrible selling point. I've seen this thinking appeal to misguided PHB-types and witnessed the result: immense organizational damage. In my experience, this isn't a necessary or sufficient selling point to good developers. Learning a new language just isn't that hard, and a big part of a shift like this is actually in learning the new environment's paradigms, APIs, and best practices.

To make the latter point more strongly: if you're having doubts about your ability to pick up a new language, definitely take some time to learn a few new languages. Do a tutorial, play with a few small projects, enough to get the flavor of the language. Your hackery will benefit immensely from this, even when you return to your primary language.

I was curious, so I actually ran both of the servers from the article on my little MacBook Air. The results are below.

First, go:

    $ ab -c 100 -n 10000 http://localhost:8000/
    This is ApacheBench, Version 2.3 <$Revision: 655654 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/

    Benchmarking localhost (be patient)
    Completed 1000 requests
    Completed 2000 requests
    Completed 3000 requests
    Completed 4000 requests
    Completed 5000 requests
    Completed 6000 requests
    Completed 7000 requests
    Completed 8000 requests
    Completed 9000 requests
    Completed 10000 requests
    Finished 10000 requests


    Server Software:        
    Server Hostname:        localhost
    Server Port:            8000

    Document Path:          /
    Document Length:        1048576 bytes

    Concurrency Level:      100
    Time taken for tests:   10.085 seconds
    Complete requests:      10000
    Failed requests:        0
    Write errors:           0
    Total transferred:      10489017384 bytes
    HTML transferred:       10487857152 bytes
    Requests per second:    991.62 [#/sec] (mean)
    Time per request:       100.846 [ms] (mean)
    Time per request:       1.008 [ms] (mean, across all concurrent requests)
    Transfer rate:          1015729.90 [Kbytes/sec] received

    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        1    2   0.8      2       6
    Processing:    21   99   5.6     98     137
    Waiting:        1    3   2.7      2      41
    Total:         25  101   5.6    101     139

    Percentage of the requests served within a certain time (ms)
      50%    101
      66%    102
      75%    103
      80%    103
      90%    105
      95%    106
      98%    108
      99%    112
     100%    139 (longest request)

Secondly, node.js:

    $ ab -c 100 -n 10000 http://localhost:8000/
    This is ApacheBench, Version 2.3 <$Revision: 655654 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/

    Benchmarking localhost (be patient)
    Completed 1000 requests
    Completed 2000 requests
    Completed 3000 requests
    Completed 4000 requests
    Completed 5000 requests
    Completed 6000 requests
    Completed 7000 requests
    Completed 8000 requests
    Completed 9000 requests
    Completed 10000 requests
    Finished 10000 requests


    Server Software:        
    Server Hostname:        localhost
    Server Port:            8000

    Document Path:          /
    Document Length:        1048576 bytes

    Concurrency Level:      100
    Time taken for tests:   15.765 seconds
    Complete requests:      10000
    Failed requests:        0
    Write errors:           0
    Total transferred:      10487558651 bytes
    HTML transferred:       10486808576 bytes
    Requests per second:    634.31 [#/sec] (mean)
    Time per request:       157.653 [ms] (mean)
    Time per request:       1.577 [ms] (mean, across all concurrent requests)
    Transfer rate:          649639.92 [Kbytes/sec] received

    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    1   1.7      1      11
    Processing:     2  156  34.7    159     272
    Waiting:        1   47  29.7     42     136
    Total:          2  157  34.7    161     273

    Percentage of the requests served within a certain time (ms)
      50%    161
      66%    174
      75%    182
      80%    187
      90%    198
      95%    209
      98%    221
      99%    227
     100%    273 (longest request)

Not only does go serve the traffic more quickly, but it also has a much lower standard deviation between slow and long requests. Impressive.

I'd be interested to see your numbers benchmarking with `siege' rather than `ab'.

wrk or weighttp are my preferred tools.

I really love 'siege'. I wonder if you can get it show you response-time percentiles (or whatever they're called). I mean this (from `ab`):

    Percentage of the requests served within a certain time (ms)
      50%    161
      66%    174
      75%    182

Benchmarking on macbooks is often an exercise in testing the mediocre default configuration of the network stack, not your language. My macbook pro gets 4k rps with apache, node, go, and nginx. YMMV and all that, but I'm always wary.

What configs do you change?

I usually just benchmark in a production environment.

Clarification.

What did you change to get the claimed 4k rps on your Macbook?

Upgraded from air to pro ;)

check the memory usage. When I did the comparison, the node server's memory climbed steadily up to about 50MB, while the Go server's memory stabilized at about 17MB.

r u using x86 go? or x64 go?

Unless he went out of his way to use x86, I can't imagine he's not using x86-64. x86-64 has traditionally been the arch that Go has gotten the most attention on anyway, it is what everyone uses these days.

In the comments of the article are some test results too, only that on that test, nodejs was slightly faster.

It'd be interesting to throw Erlang into the mix, too.

One example:

http://eric.themoritzfamily.com/websocket-demo-results-v2.ht...

Has Node.js, Erlang, Scala, go and a lot of Python. On EC2 Erlang and Java pretty much blow everything else out of the water.

I did this test a while ago on my laptop running Linux Mint. With higher concurrency levels (IIRC, about 1k simultaneous connections), I saw just the opposite, with cracks starting to show and Go performing worse than Node, although the V8 engine did consume more memory.

fyi, in the replies of the original post someone got totally different results. with v0.6 it was significantly faster than Go and with 0.8 was on the same level

I run node/express for most of my web servers and each takes up about 10-15mb RAM. They're very basic no fluff. Anyone know what comparable mem footprint in Go?

Here's a quick paste from a server of mine:

    ps aux | grep api
    ubuntu   15720  0.0  0.1 107024  6136 pts/0    Sl   15:13   0:00 bin/api_server

IIRC ps displays the resident size in KiB, so you're looking at 6 MB for a Go process. Not terrible for a high-level language.

Yep, just ran ps_mem.py (http://www.pixelbeat.org/scripts/ps_mem.py) and here's my entire server. Not a busy one, but it lets you see what a running api server and accompanying programs look like.

    Private  +   Shared   =  RAM used	Program

    184.0 KiB +  31.5 KiB = 215.5 KiB	atd
    240.0 KiB +  55.0 KiB = 295.0 KiB	cron
    240.0 KiB +  68.0 KiB = 308.0 KiB	upstart-socket-bridge
    304.0 KiB +  72.0 KiB = 376.0 KiB	upstart-udev-bridge
    392.0 KiB +  79.0 KiB = 471.0 KiB	sudo
    696.0 KiB +  26.0 KiB = 722.0 KiB	dhclient3
    604.0 KiB + 189.0 KiB = 793.0 KiB	getty (6)
    940.0 KiB +  49.0 KiB = 989.0 KiB	dbus-daemon
    660.0 KiB + 366.0 KiB =   1.0 MiB	udevd (3)
    1.0 MiB +  71.0 KiB   =   1.0 MiB	rsyslogd
    1.1 MiB +  35.5 KiB   =   1.1 MiB	redis-server
    1.0 MiB + 122.5 KiB   =   1.2 MiB	init
    964.0 KiB + 733.0 KiB =   1.7 MiB	polkitd
    1.4 MiB + 823.5 KiB   =   2.2 MiB	console-kit-daemon
    2.5 MiB +   1.1 MiB   =   3.6 MiB	nginx (5)
    1.3 MiB +   3.2 MiB   =   4.6 MiB	sshd (5)
    5.4 MiB +  75.5 KiB   =   5.5 MiB	api_server <==== Go Program
    13.8 MiB + 963.0 KiB  =  14.7 MiB	bash (2)
    22.5 MiB + 314.0 KiB  =  22.8 MiB	mysqld
    41.9 MiB +   5.1 MiB  =  47.0 MiB	python2.7 (2)
    79.4 MiB + 542.0 KiB  =  79.9 MiB	java
    ---------------------------------
                        190.3 MiB
    =================================

I find this post paired with this thread confusing. Yes, Go is tempting and I'd like to try it since a lot of people get quickly into flow with Go, the "package manager is so great" and "everything is just a breath of fresh air".

But what I don't like: the negativity against Node and omitting some facts. In the replies of the orignal post a guy tested two (!) times Node and once it was significantly faster (v0.6) and once it had same speed (v8.0). So, why has mjijackson such different results in this thread at the top?? And maybe we should test it on real servers and not on a MBA. Moreover, we have here some micro benchmark which possibly doesn't reflect reality well. Don't get me wrong, I appreciate any benchmarking between languages but then please do it right and make no propaganda out of it. Further, Go's package manager seems to be nice but it does NOT have version control. How do you want to use this in a serious production environment. Maybe version control will come (but then tell how without loosing its flexibility) or not but this is something serious and definitely not an alternative to any server environment except for some mini services.

EDIT: downvoting is silly, propaganda and won't help the Go community in getting more credibility, better do some further benchmarks; otherwise this post/thread is full of distinct misinformation and should be closed

Go's package manager does have version control. It looks for specially named branches (different ones depending on the version of go you have). The upstream authors can provide a different version of the software for different releases of Go.

If you want to lock down the versions of all the software you're deploying in your organization, that's easy to do too. Just "git clone" all of the libraries you use to some internal server (and/or github repos), and change the URLs to point to that server. You control when everything gets updated.

Golang builds static binaries anyway. So if you test a binary and it works, you just copy it to all the servers you care about and you're done. If you're in a small and informal shop, maybe you don't need to mirror every repository. Due to the nature of git, if the upstream repo ever gets deleted, you can fall back on a local copy of it anyway.

This is all very much in contrast to languages like Java where keeping around the proper version of every jar and carefully deploying that version (and only that version!) on each server is big deal (and despite OSGI, still very much an unsolved problem.)

The go import tool does version go std lib packages, but I think the worry is more about external packages - later when the go ecosystem matures, and I'm relying on lots of packages from different sources (say for a web app), with interdependencies, and I must update them regularly for security updates, but don't necessarily want the latest master for all, I have no way to specify versions without rolling my own personal package management, downloading the packages and changing urls etc. If I'm in an org and sharing this with others I'd basically end up reinventing a package proxy internally in order to control what software versions are used to build.

This is a small flaw in the go packaging system (and it is a flaw) which I'm sure they'll fix, either with a convention to always use branches for versions (which I guess is doable, but only if it becomes widespread), or by changing import to take an argument for the version. So something like:

    import "github.com/gorilla/mux/1.0.3"

    import "github.com/gorilla/mux", ~> "1.0.3"

which would let you specify any minor updates say and possibly other permutations, and thus might be more flexible. I believe a few things have been suggested on the list but I haven't followed the conversation, not sure what the outcome was - perhaps just that it needs further thought.

At present the first solution is possible, BUT it needs to become a convention which everyone follows in order to be useful (involving named branches or tags at the repo). The other advantage of this is that it states requirements fully in the package concerned - otherwise all we know is that this code requires github.com/gorilla/mux, not which version or when it was last tested/imported, whether it will work with the latest release or not, etc. But then it would let you run into conflicts by accidentally importing two versions of the same package with different paths.

The current convention of no version certainly puts more onus on the package maintainers to maintain backwards compatibility, or on users to maintain their own library of packages which they periodically update, and I see the arguments for it.

Of course none of this matters if you're just one person using packages to build an app at one moment in time, but if you yourself supply packages/apps to others, or want to setup members of an org over time with the same set of packages in the same state, and rely on other packages to compile yours, it can become more complex.

Other package managers tend to have a central point to adjust package dependencies, and make sure packages are always available forever at the same uri, so it'll be interesting to see how go manages this when go packages become more complex and widely used, start to be removed by maintainers, and start to have complex chains of dependencies on specific versions, or whether the much simpler go system in the end leads to a saner situation and puts the onus for this problem back on end users.

Great having one bringing a balanced view on Go and showing that Go is still not ready for production when used with external libs. A refreshing take in this overhyped thread.

I'm not sure I'd agree that it is not ready for production, but there are some kinks (relatively minor I feel) which will become more apparent when it is used more widely than it is currently in production. Just as Ruby (for example) has gone through a trial of fire since becoming popular because of Rails, and has been much improved because of it.

Java and C++ never developed package management at all, but have been used in production for years. Golang is already ahead of those languages here.

it'll be interesting to see how go manages this when go packages become more complex and widely used, start to be removed by maintainers, and start to have complex chains of dependencies on specific versions

"Complex chains of dependencies on specific versions" is an anti-pattern, common in the Java world. I am glad that Golang doesn't support or encourage this anti-pattern.

If someone deletes your favorite package, then you can just re-upload it from your cache and let people know about the new URL-- assuming that the code is open source.

If you want to make a backwards-incompatible change in your library, then just create a new version and call it something (slightly) different. There's no reason to complicate things.

"Complex chains of dependencies on specific versions" is an anti-pattern, common in the Java world.

I do have some sympathy for this radical simplicity which Go aims for, but at times it comes at a cost, and at times a little bit of complexity has to be added to deal with the real world (only very rarely and after much thought though). I'm torn between admiring that they don't rush into new features and impatience with a few small things they've refused to add so far.

If someone deletes your favorite package, then you can just re-upload it from your cache and let people know about the new URL-- assuming that the code is open source.

Well yes, but then you're de-facto maintainer of that package, you might not have permission to do that, etc - this point though I feel is less important than versioning, it's more an argument for central package control, which has pros and cons. I'd be happy if Go never does that and agree that it can be worked around but will be interested to see if people judge it necessary eventually. There's nothing to stop people setting up a central package resource, and nothing really needs to be added to go to support it.

If you want to make a backwards-incompatible change in your library, then just create a new version and call it something (slightly) different. There's no reason to complicate things.

This is not an acceptable solution, it would lead to situations like:

    import 'github.com/user/yaml_parser'
    import 'github.com/user/new_yaml_parser'
    import 'github.com/user/really_new_yaml_parser'
    import 'github.com/user/toms_new_yaml_parser'

What a mess, particular for new users trying to decide which package is best or canonical. Why not just use versions?

I wouldn't be so quick to defend the status quo - Go is a young language with plenty of maturation still to do, they could easily add versions to packages if it proves necessary - I suspect in the long term it will, because large ecosystems do involve chains of dependencies (I'm not talking about java here, but perl, python, ruby, java, C++, etc all of these languages version libraries in some way), and developers of packages make mistakes which they need to fix, sometimes by deprecating or removing compatibility, but their users don't want to deal with those mistakes, sometimes for years, and the users of their users certainly don't. Choosing a new name and orphaning all your users is not an acceptable solution.

I'd prefer if package maintainers could just do this:

    import 'github.com/user/yaml_parser'
    import 'github.com/user/yaml_parser/1.0'
    import 'github.com/user/yaml_parser/1.0.1'

etc and left the master as the plain name with version tags as longer names. It'd be nice if go supported this by supporting arbitrary branches or tags on github etc but I don't think it does at present, happy to find out I'm wrong.

If you're using a library from github (or wherever) that isn't open source, and someone takes it offline, that's a legal problem, not a technological one. If you are using an open source library, then just re-upload it (or find someone who will). I don't see why the language has to do anything.

What a mess, particular for new users trying to decide which package is best or canonical. Why not just use versions?

You know what a mess is? A mess is:

* In a Maven build, A depends on version 1 of C, A depends on B, B depends on version 2 of C. Result: Java crashes at runtime.

* People using ancient (potentially insecure) versions of Java packages for years because they're afraid of the previous problem.

* "deprecated" functions that continue to be used for years, spewing warnings each time they're referenced.

* a time-consuming bureaucratic process to get your library into some "blessed" central repository.

Go doesn't "support" these messes, and I see no reason why it should.

Perhaps Go's dependency management system will change. I can't tell the future. But so far, I haven't seen any good arguments in favor of changing it. All of the problems you think you're solving with versions could be solved by either maintaining backwards compatibility, or choosing a new package name when you feel you have to break it.

One advantage of node that wasn't mentioned is the ability to share server side and client side code. Avoiding discrepancies in the same form validation written in two different languages can often be more important than performance gains in server applications.

>One advantage of node that wasn't mentioned is the ability to share server side and client side code

You can do that with lots of languages, using one of the langX->javascript compilers.

>Avoiding discrepancies in the same form validation written in two different languages

What form validation are you doing in javascript? I can't actually think of any client side validation that isn't just a regex.

>There are also some officially maintained repositories outside of the stdlib that deal with newer protocols like websockets and SPDY.

Does anybody from HNers use Go with websockets? What package do you use?

What do you use for persistence? I know what's available, but which database and library could be recommended for a low to medium traffic web site?

How long does take the compile time with a medium sized typical web project. Something which gets annoying or still bearable?

I am working on a project that has a few thousand lines of Go and compilation takes a second or so. Maybe less; it's fast enough that I don't really think about the fact that it's compiling unless I've done something that it complains about (which is awesome).

If you mean go, compile time is very fast, even with say 10 kloc you'll still probably be under a second.

Is there a socket.io equivalent for Go? In terms of ease of use and at least providing the same functionality.

Looks like there are a couple packages that would solve this problem, but neither appear to be production ready or up-to-date.

https://github.com/madari/go-socket.io

https://github.com/igm/sockjs-go/

I wonder how Go compares to SilkJS, since SilkJS is much faster than Node.

SilkJS uses mostly the same libraries and relay on the same VM as Node.js: V8. So it can't be much faster. The note how it outperform Node.js http server, you can read from start page of its github repository, are misleading since it uses multiple processes (SilkJS http server forks itself).

Reply on news.ycombinator.com