Trying out this Go thing at Disqus
blog.disqus.com
blog.disqus.com
Unfortunately, there is no technical content (which seems endemic to the recent storm of Go articles). Is it faster because is a static language producing machine code? Was their choice of Python packages wrong? How much are the performance improvements caused by knowing better where the bottlenecks are? Were other languages with larger ecosystems considered?
Internally, we had each played with Go for a little bit, and felt very natural to us. There weren't any other considerations really.
For comparison, we're still dark writing to the old realtime just to make sure that nothing is broken, and old realtime is consuming easily 16x resources and technically doing a bit less since it's not really publishing anymore.
"... and then it's tempting to start kludging. I have a suggestion: get it right the first time."
Whenever I'm programming I can still feel his disappointed stare of disapproval when applying a quick and dirty hack.
EDIT: And of course, there's learning from your mistakes so you can get it right on the first try the next time you have to do a similar thing.
Let's see Harm Bakker "get it right the first time" under changing conditions, sloppy inherited code, a pointy haired boss, etc...
Where this falls down in the real world is that oftentimes your understanding of the problem is quite limited at first. You'll ship a demo or an MVP, and that will teach you a little. As you continue to explore the problem through your work you'll often redefine it. Or it will redefine itself, as happens in so many war stories of service scaling, product maturation, and so on.
Is it really a surprise that the Python VM is not performant? I thought everyone knew that by now. Any reasonably well engineered VM (e.g. V8, JVM, Go) will blow it out of the water. This should be well known to anyone who calls themselves a software engineer.
The Disqus system could have been written in a large number of other languages and have performed more or less the same as the Go system. The only advantage I can see of Go is that it's reasonably close to Python/Ruby in terms of semantics. If Python or Ruby is your only tool then Go might be a good choice. Not so if, for example, you're more into the functional style.
Edit: Ok, so Go doesn't use a VM. It still has a runtime, as all languages must. Substitute runtime for VM above as appropriate. Point still stands.
FYI, Go does not use a VM like V8 or the JVM; Go code compiles to native binary.
However, like many VM based languages and unlike (a lot of) C, Go should be write once, (compile) and run everywhere(Win,Lin,BSD,Mac). Also like many VM languages Go is type/memory safe and is garbage collected.
Edit: It appears V8 is also not a VM, it is a interpreter that performs just-in-time (JIT) compilation of source with no intermediate representation such as bytecode (Java instead JITs bytecode). This makes sense since Java programs are distributed as bytecode and Javascript programs are distributed as source. @calinet6: Thanks for pointing this out.
Life-cycles:
Developer's machine | Runtime
--------------------------------------------------------
Go: source->binary | exec binary
Java: source->bytecode | interp. bytecode / JIT & exec bin
JS: source | interp. source / JIT & exec bin- started coding after Java and .NET became mainstream
- only know C and C++ as compiled languages, never tried anything else (e.g. Modula-2, Delphi, Haskell, ...)
- lack CS background
So they think any language with stronger type system than C or C++ can only be provided via a VM based runtime.
So in that sense, yeah, V8 is not really a VM but just a JIT compiler/interpreter.
Then try PyPy
It's OK to be wrong, just don't be a jackass about it... people will take you a lot more seriously if you're gracious when they notice it.
The CPython implementation is very slow. There are many language implementations that are much faster (e.g. O'Caml, Haskell, Scala, Clojure, Java, Lua, C, probably Rust). Any one of these is likely to produce a system about as fast as the Go system with about the same amount of effort. Thus dwelling on Go being faster than Python is not interesting.
What is interesting is the properties of Go that make it better or worse suited to particular organisations and problems. This is what I tried to get at in the second paragraph.
The precise implementation strategy of Go/Python/V8/whatever is interesting in its own right but irrelevant to my points.
There is nothing special about ditching Python and getting speedups. Just last week I did the same with rewriting a service in Erlang - and I'm almost certain that it will perform even better on 8 cpus than even Go would.
...and dropped a link (Thoonk) in the processing chain...
"I didn't know anything about Go other than 'it sounds cool', so I decided to rewrite this very important bit of Disqus' infrastructure. In one week, from start to finish, I had it implemented, tested, and deployed and improved latencies by several orders of magnitude and reduced the capacity requirement from four fully loaded servers to 20% of one!"
To be able to say that is pretty impressive, worth writing about, and definitely interesting to me as someone who wants to see Go used more. Disqus is well known, has lots of traffic, and so a recommendation from them carries a lot more weight than some random benchmarks. (As cool as they are.)
Maybe an insider could shed some light on this campaign. Does Google marketing dedicate people to cater for social networking sites and news aggregators? How is the 'storm' organized? Obviously, many startups try to spread the word through 'cheap' channels. Only few are really successful.
For what it's worth, outside of the FAQ, I don't think golang.org even mentions Google. In general, Go is kept very distinct from Google, so it wouldn't make much sense for Google to embark on a Go marketing campaign.
2. You don't have many options aside from Gevent.
3. I can't talk for them, but my Python bottlenecks were mainly memory related. I was simply using up too much of it, due to how it was handling each request. My pattern was a basic Data/Handler/Json one. Which is as simple as they come. Yet, I was having issues (at a small scale).
4. Yes, of course. I considered and built parts using other languages. Did some testing, too. But Go stood out as the best choice given my needs.
I haven't gotten to the point where I've looked into things like web frameworks, templating engines, Postgres adapters, anything like that. But I'm interested in it.
For a heavyweight web app that needs to scale to multiple machines I would've done things differently. It turns out that Go is so fast that needing to scale would be a luxury problem for me. In theory I can serve 70M users spread over a 10-hour day for < $100 a month.
Nginx as a front facing server. Go listening to a specific port. MySQL for general database operations. MongoDB to pass some data around.
Shoot me an email. I'll get you up to speed quickly.
Sent you an email
With Go you don't have anything comparable to Django, Numpy, Pandas...
Given the age of the language I would assume building out its associated toolchain is only a matter of time. Python didnt ship with Django, Numpy, Pandas...
Basically Python is the glue language data people wanted, it turns out.
Will it replace all of our Python? Highly doubt it. Some very specific pieces? Probably.
But to answer your question, the Language Shootout seems to suggest Java and Go are on the same plane in terms of speed. Take that with however many grains of salt you like.
Clojure's done some impressive improving, too.
Nimrod? Never heard of it or the person backing it.
I'm _not_ a language snob, trust me! I'm just a regular, family guy, software developer and I try to put my proverbial eggs in reliable baskets.
Go doesn't seem to be going away any time soon and it's really really fast.
I chose Go over C++, because it looks easier.
C++ is harder and there's no garbage collection.
I now have to yet again close my apps and restart my computer for an IT-forced Java runtime update, for some app I rarely use. For that reason alone I'd not consider Java.
But I don't want you to use C#/C++/Java, I want you to use Nimrod.
This is the first I've heard of Nimrod.
Beyond that, it deals with a lot of the 'ugly' things around the edges of other languages. Dependency management, build management, deployments... all these IMHO are much more well thought out in go.
I have to disagree on the concurrency model, I think message passing channels are a much more natural primitive to model concurrency in, and goroutines are exceptionally light.
EDIT: When you talk about nimrod, you might go ahead and mention you are the designer of nimrod... it might color your judgement.
Go's primary niche is server software, and in that niche, it is gaining in popularity and has the backing of a large company. For servers, neither support for a 32-bit address space nor real-time support is important.
Does support for generics really matter when the language has built-in support for the most common collection types?
The allowance of shared mutable memory between goroutines does worry me somewhat.
This entire project was also stood up in a week, so it's not like this service is some monster service with a lot of work invested into it. It was small enough that instrumenting and figuring out where the problems may lie and figuring them out would have probably taken just as much time.
However, I suppose without details of the service itself, it's hard to make a judgment call from the sidelines.
Still, glad you were able to improve performance. Why was 'Go' chosen?
Nevermind, I see that was answered in another thread.
All the articles and talks I see on scalability and optimization use production servers as examples, but as you way, you want to optimizing for traffic instead of responding to it.
You'd expect very nonlinear behavior between number of users, response time and number of machines used.
Not being a part of Disqus, the only way I see to "predict" performance beyond the load already seen is to try and use past data from the current real system.
Erlang is memory safe in the presence of many-core (or many-system) parallelism. Go is not (you can segfault, possibly in an exploitable way, if GOMAXPROCS > 1).
Erlang unbounded channels reduce deadlocks because your sender can continue execution without waiting for a receiver.
What's also true and potentially compensatory is the C-like degree of control Golang gives you over how you allocate memory and lay it out.
I'm not sure the unbounded channel thing is a real advantage for Erlang. I'm happy to be convinced I'm wrong. What's a real, correct design which would be hard to realize in Golang (without unbounded channels) that relies on unbounded channels?
In general most Go apps that have been deployed are Web apps and server infrastructure, where concurrent garbage collection is not too much of a problem in practice. So Go's choice makes sense in Go's context. It does limit parallel scalability in some contexts—which of course are not the contexts that most people have been using Go for at this point.
> What's also true and potentially compensatory is the C-like degree of control Golang gives you over how you allocate memory and lay it out.
Go doesn't give you C-like control over allocation of memory. Language constructs will allocate memory in ways that are not immediately obvious, to quote Ian Lance Taylor [1]. It does give you control over layout of memory.
> I'm not sure the unbounded channel thing is a real advantage for Erlang. I'm happy to be convinced I'm wrong. What's a real, correct design which would be hard to realize in Golang (without unbounded channels) that relies on unbounded channels?
Suppose you're pulling down images from the network and printing out a sorted list of URLs of all the images you find. You might structure it as two goroutines A and B. Make two channels, "urls" and "done". Goroutine A is the network goroutine and simply crawls looking for images to stream to B over the channel "urls". When it's done it sends "true" on "done". Goroutine B is the sorting goroutine and first blocks on the channel "done" before it proceeds, after which it drains the "urls" channel and sorts the results.
This program contains a deadlock due to synchronous message sends. If there are more URLs to be downloaded than the buffer size of "urls", then the program will deadlock. If "urls" were an asynchronous channel, however, this would be fine.
Of course this can be structured to fix it, by doing the send in another goroutine for example (although that costs performance). But hopefully that's a good illustration of the subtleties of synchronous message sending.
[1]: https://groups.google.com/forum/#!msg/golang-nuts/LoJJ1bICdu...
These are two sides of the same coin. Having control over memory layout allows you to implement what are in effect allocators.
(Also, if you implement memory pools, the GC still has to trace the pointers within at mark time.)
Here, you will never actually block on your send since it runs on it own goroutine. I can't see an actual use case for this kind of thing but since you are using this argument over and over then.. :)
In general I consider segfaults exploitable, because of heap spray and virtual method calls. Even if not "exploitable", I consider it "very very scary".
Your point is a reason that Golang couldn't be dropped in as a browser Javascript replacement. But nobody has ever suggested that it could be; it can't, just like Java (which was designed for the purpose but failed at it) and Erlang (which wasn't) can't.
What if a Go program allowed Go objects to be scripted by untrusted user code written in JavaScript? In browsers it is very possible to corrupt the Frame (Gecko)/RenderObject (WebKit) tree, which is in a C++ heap that is separate from the JavaScript heap.
The memory safety issues in C++ are a problem not because they mean that untrusted code written in C++ can't safely be executed (although it does mean that). It also means that safe languages become easily weaponizable. JavaScript (or Lua, or whatever) embedded in a Go program could paint the heap with malicious addresses.
The Javascript/C marriage problem isn't "heap spraying", it's that the interpreters themselves are full of exploitable C bugs, which is made much worse by the fact that the Javascript object lifecycle is expressed in an inherently unsafe language and so every tuple of [reference, event] has to be diligent checked. The same simply wouldn't be the case for any realistic marriage of Golang/Javascript, if only because the number of exploitable code conditions in Golang is miniscule compared to that of C.
All heap spraying does is make bugs that are very plausible to exploit easy to exploit reliably. You still have to start with "plausible".
I'm not confident that race conditions in a shared-everything language are not "plausible". My experience is that race conditions are subtle and hard to find, even with a race detector. All you have to do is race on a map or a slice. And virtual calls are everywhere in Go.
Sure, we don't know that it's a problem so far, as nobody has created such a scenario. We're in violent agreement there. I grant that for server-side use cases, it doesn't matter—people use enormous C++ server codebases in production all the time and memory safety issues rarely bite them to the same degree that we see in browsers.
All I'm saying is that I don't have the same level of confidence that Go is free from memory safety exploits that I have for, say, Erlang or Java.
You're wildly off the mark when you say that C++ serverside code tends to survive against attackers looking for memory corruption bugs. They do not. They fail with memory corruption flaws routinely. That was a cheap shot (you tried to create an equivalence class of unsafety between two totally unrelated languages and two totally unrelated sets of bug classes) and it won't work. You're going to have to try harder to make a case, if it's worth it to you.
Nothing is as bad as browser Javascript (it would be hard to conceive of a harder software security problem to design against), but C++ server software is pretty far towards the "unsafe" side of the security spectrum, and Golang and Erlang probably occupy virtually the same spot on that spectrum.
I've described a scenario whereby a Go program that embedded untrusted safe code could fall to memory safety vulnerabilities. To be exact, it creates a slice of interfaces and accesses the slice in a racy way in such a way that it calls virtual functions at the same time it inserts, causing the slice to be reallocated. Then an attacker sprays the heap with addresses of shellcode. This results in arbitrary code execution when calling a virtual method.
You're saying that this is so unlikely as to be implausible, as it's never been observed in practice and might not even work. That's fine, I respect that position. We'll leave it there, and agree to disagree about whether this is a concern relative to languages like Erlang that are designed to be 100% free of memory safety problems. :)
I'm not trying to be pissy about it; I make bogus arguments all the time too, often without realizing it. You obviously know what you're talking about. I just think in this one subthread, you're wrong.
But the data races don't bother me at all, first of all you won't encounter them if you embrace a program design facilitating goroutines and channels (the often quoted "don't communicate by sharing, share by communicating"), and second because we now have the tools to detect data races.
* Golang has a more conventional, familiar, boring, "safe" language design than Erlang does. If your current language is Python, you're apt to find Golang congenial; I describe Golang to my friends as "a modernized hybrid of Python and Java".
* Golang has a more flexible concurrency offering than Erlang. It has a very flexible (statically typed, sync-or-async) first-class channel type that you can deploy anywhere in a program whether you parallelize it or not. More importantly, it has first-class support for "conventional" shared-everything concurrency with mutexes and semaphores and whatnot. If you're designing everything from scratch to fit your language's preferred idiom, this may not matter, but if you're porting an existing design it might matter a great deal.
* Golang has what I will perhaps hyperbolically refer to as first-class support for strings, with reasonably performant regular expressions, a clearly delineated "just a bag of bytes" type designed from scratch for high-performance buffering and packet framing/demux, and a sane approach to UTF-8 Unicode. It also turns out to be a very elegant framework (as imperative languages go) for building parsers, since you can put the concurrency stuff to work for it. These are common programming tasks and not one Erlang is well known for handling gracefully.
* As I understand it, both Erlang and Golang have comparable lightweight process/thread/coroutine/whatever facilities. Golang was designed from the start to handle huge numbers of demand-spawned threads with small, dynamically allocated stacks. I'm not saying Golang does this better than Erlang does, but it might be tricky to argue that Erlang's process model is better than Golang's.
* Erlang requires an installed runtime and executes in a VM. Golang produces native binaries. You can compile a Golang program on one machine, rsync that single binary to another machine that has never been blessed with a Golang installation of any sort, and run it without a problem.
* Golang has modern, extremely high-quality tooling. Entire huge Golang projects rebuild in seconds or less. The compiler is extremely helpful. Code formatting and documentation are first class problems for the toolchain. The Go package system is well thought out, and, again, is familiar to Python programmers.
Just some guesses as to reasons in there.
This is a taskbar button press for me.
> The compiler is extremely helpful.
Agreed, and for runtime errors Go shows the entire stack with accurate line numbers and good error messages.
I know there are tools like Chef out there, but I can spin up production-ready Go application servers on any Linux box in a minute because of this feature.
Erlang shows, logs, send via mail and telepathically tells you about what died where with all the details, and then additionally restarts the failed process according to restarting policy and tries again. Or not.
Anyway, I think this thread was about comparison of Erlang and Go - other threads are already full of people writing how wonderful Go is. As a part time Erlang developer, in this thread, I'd like to read what Go has better than than Erlang, not what is good in Go in general, because the latter is just increasing noise-to-ratio.
And it's possible to build single binary and deploy it with Erlang too.
Static typing with a very convenient expression in the language.
A concurrency model that is easier to adopt if your frame of reference is concurrent C++.
A simpler, friendlier syntax, which is probably not a win if you're a veteran Erlang programmer.
Probably better tooling: native binaries, a lightning fast compiler with great error messages and testing facilities, &c.
Perhaps a more modern standard library, which is made somewhat simpler and more concise by the pragmatic adoption of a very little bit of conventional OO, without going whole hog the way Java does.
There are certainly advantages to Erlang, too!
Erlang processes have very small stacks just like goroutines do.
When people talk about asynchronous channels, they usually mean that you can stream messages to another actor and know that you won't block. That is not true for Go channels. You can increase the buffer size, but that just reduces the chance that your program will deadlock: it doesn't make a Go channel work in situations where you need an asynchronous channel for your program to be correct.
go func() { ch <- foo }()
For what it's worth, I don't think that Go made a bad decision here (although I personally wouldn't have made the same decision, because of examples like those I gave in my other reply downthread). Certainly synchronous channels are faster. There are always tradeoffs.
I'm using both, Erlang/OTP and Golang. Both have their strengths and advantages. So simply use the right tool for the right work (and no hammer to turn a screw).
Do you mean "safe" as in "nobody got fired for buying IBM" or as in language making hard to start thermonuclear war by accident? Because Erlang, with it's immutability everywhere is as safe as, or safer than Go.
> Golang has a more flexible concurrency offering than Erlang.
Basically everything you wrote here Erlang has. And what they were porting was in Python + GEvent, which means no threads anyway.
> a clearly delineated "just a bag of bytes" type designed from scratch for high-performance buffering
Erlang's binaries and strings (two different types) are designed for the same thing.
> but it might be tricky to argue that Erlang's process model is better than Golang's
Well, Go did a nice job borrowing Erlang's solution here. As it origins from Erlang, it has to be good. ;)
> Erlang requires an installed runtime and executes in a VM. Golang produces native binaries.
Erlang is able to produce a single binary. That it contains a VM and libraries and user code is another matter, but rsyncing Erlang binaries is also possible.
> Golang has modern, extremely high-quality tooling.
Dialyzer, EDoc, various process viewers, debuggers and so on for Erlang are really mature.
I think that familiarity of syntax, familiar programming paradigm and raw (single threaded, number crunching) speed were the reasons. Go has many benefits over Erlang in the realm of marketing, but I don't believe Erlang is much worse a language because of it.
But then, you cannot have it both ways. Either Erlang is "safer" because of share-nothing and immutability, or Golang is more flexible w/r/t concurrency. I can design a shared concurrent data structure in Golang that is protected by locks or atomic counters, and I can do that easily, using the native first-class facilities of the language. Can two Erlang processes cooperate using a single shared buffer with a custom-designed concurrency scheme? Is that a natural thing to express in Erlang? It's a natural thing to express in Golang.
I feel like you didn't really engage with the string processing point I made.
I feel like it doesn't help an Erlang advocate's point too much to observe that Golang stole the best parts of Erlang.
I feel like every time it is ever pointed out in any language comparison that Golang can produce runnable native binaries, someone always has some rube-goldbergian alternative that nobody in the real world ever uses to get their own language bootstrapped onto some routine from a single file.
I gave specific reasons why Golang's tooling is strong. I feel like I got a response that says "Erlang's tools are mature". Nobody is arguing that Erlang is immature. The argument is that it's geriatric. :)
Ultimately, I agree that most of what I think is good about Golang is probably icing on the cake after "the language is boring enough to be familiar to Python programmers".
There's probably no match for Erlang's error handling mechanisms out there, I actually dislike Go's panic mechanism, but if you have a small piece in your system which needs to be wickedly fast and highly concurrent I see Go as a perfectly good choice.
As always your program is the best benchmark.
I have to say I always thought Disqus were a cool team, I have been doing websites for over a decade and whenever I needed a comment system they were the first choice. It was a real shame when I came up against their recent decision to push out the 'Discover' feature. It resulted in my having to remove Disqus from most of my clients websites as I received several angry emails complaining about porn links (it wasn't porn, but sleazy celeb stuff like bikini photos etc). Sad to say I have written them off as a commenting platform; what they did, although probably a brilliant business decision and making them loads more ad money, felt like a complete betrayal of trust.
How does GC impact that? What about C to Go inter-op. Is there a latency penalty?
Are you referring to the cgo stuff?
C inter-op speed would be a big part of any such claims as any systems language would need to support that.
That said, go actually can come close to C on sustained throughput. (On some benchmarks, naive go programs beat naive C, but usually go is somewhat slower.) You can interoperate with C using cgo, and there is no penalty. (It may not work with all go compilers.) If you want to interoperate with C++ then see http://www.swig.org/Doc2.0/Go.html for the best solution. I believe that there is a performance penalty there.
Like C, Go is compiled to a native binary. Certainly something like GCC will tend to produce faster code, but there's nothing fundamental here preventing you from writing code that can run fast in Go.
The real question isn't "can go achieve C latency" but rather "will every-day code written in Go achieve the same latency as every-day code written in C". And the answer is probably: it'll be slower, though usually in the same ballpark, but it will be easier to write, easier to maintain and have fewer bugs.
If Go can't do that (or can't do that yet) it's fine by me, I'll just let it bake a few more years and reassess, but there have been tons of articles lately talking about how great Go's performance is. I'm just curious if that is an artifact of the original languages being compared to, or if it has broken through to C levels of latency. If it has, it becomes a lot more interesting to me.
http://en.wikipedia.org/wiki/Real-time_web "The real-time web is fundamentally different from real-time computing"
Seems like it would be nice if I could just say "I need this done fast, I'll use Python" or "I need 'performance' so I'll use Golang"
And to be fair, many benchmarks in Go don't perform nearly as fast as C or even the JVM. Go has gotten faster with each point release but again, these sorts of raw performance benchmarks aren't really good for anything except providing an apples-to-apples comparison. Because it's not usually a question of building the same exact app in the same exact way in one language or another.
Could that be... Resque? If so whats the problem mentioning you are using Ruby as well?
Jobs were originally pushed from our Django app, into a Celery job, which consumed the message, and published into Thoonk. Thoonk was then consumed by the old Python backend.
With this rewrite, we've also removed Thoonk and are using a fanout exchange in Rabbit directly. Go consumes the Celery job from the Rabbit queue.
Just kidding.
Before Go a bunch of shops switched from ruby and rails to scala for various reasons. Before scala companies switched from either php or something enterprisey like Java/C# to rails. Before that people switched from C++ to Java/C#. Before that C to C++. Before that ASM to C. Before that punchcards to ASM.
Thus, in terms of the hype cycle, Go is the new Scala.