Making the Switch from Node.js to Golang
blog.digg.com
blog.digg.com
Step 2 - Use it until you find its achilles heel.
Step 3 - Learn a new technology which doesn't have that achilles heel.
Step 4 - Goto Step 2.
To break this cycle, you need to appreciate that every language and library has its strengths and weaknesses. Learning what they are, and tailoring your architecture appropriately is better than jumping from one place to another every time you hit a new problem.
I can't believe the CTO of digg bought this but I guess ultimately, it's also about making engineers happy.
Eventually Go will also reach a point where it will start lagging and then some smart engineer will conclude that you need to switch to C++! Why can't developers just admit that they don't know how to scale a service?! It's not the language that's the problem, it's you; it's always you!
Maybe, but why not try Go on a small piece and see how it works.
> I can't believe the CTO of digg bought this...
You make it seem like the CTO was somehow being conned. Like Go was a bad choice. Maybe it was as simple as wanting to see what another language offered and this was a good time to try it out.
> Why can't developers just admit that they don't know how to scale a service?! It's not the language that's the problem, it's you; it's always you!
Oddly enough, this is a language feature built into Go from the get-Go. The language is designed such that a mid level programmer can ramp up and write performant Go code in a short amount of time without needing years (or even months) of experience, theory, classes, third party libraries, tooling, etc. It's a bold experiment and it may not succeed. Has it for Digg?
> Consequently, when any request timeouts happened, the event and its associated callback was put on an already overloaded message queue. While the timeout event might occur at 1 second, the callback wasn’t getting processed until all other messages currently on the queue, and their corresponding callback code, were finished executing (potentially seconds later).
Seems pretty specific to node, no? I mean, just looking at what the article says (again, I'm not a Node or Go expert) it seems like this problem was directly alleviated by Go and the speedup wasn't because Go is closer to the metal, but ultimately had a different method of handling many async network calls.
Another nit: The article also implies that the height of the flame graph would indicate a performance issue, which I argue is inaccurate: Since the height of a flame graph measures stack depth, I argue it also represents abstraction. So long as most of those stacks are are equal-width, then the abstraction is low overhead (and probably has value to justify it). Generally with a flame graph you want to be making comparisons within the graph, comparing the time spent in each call (so the relative widths); not commenting on its shape in-general (though it does always make for a pretty blog post).
Yes it is, Go is truly concurrent, NodeJS is not. Go has green threads in userland, NodeJS does not. A Go program can actually perform multiple operations at the same time since Go coroutines are implemented at the language level, NodeJS cannot. Obviously Javascript is more expressive but Go programs are easier to read and maintain. With Javascript anything goes.
Go also has "multi-process" concurrency. Go is both concurrent and parallel, Go coroutines will spawn on every processor available by default. NodeJS needs an external tool like Redis to enable "multi-process" communication. Go does not.
Go makes it more frictionless to write parallel code than in Node, but its certainly not impossible to write good parallel code in Node.
I wouldn't blame you nor the article author for not understanding the distinction - most people hear "Node is single threaded" and just accept that at face value. Coming from Ruby or Python, where the fork-process-to-be-parallel scheme is common, I am familiar with this variety of parallelism.
So the take away is Go is better at handling massive amounts of open outgoing network calls.
In Node.js you want to restrict this to prevent the server from choking. If they did that I bet performance would be very similar.
I've heard this "The Node.js event loop gets saturated" argument so many times - It's the #1 argument used in all articles which promote switching from Node.js to Go.
The real reason why the event loop is saturated is because your Node.js process is doing too much work! The solution is to split your process into two (use the Cluster module or a load balancer - It's super easy).
With goroutines and the like, all Go actually does is delay the inevitable point when you have to split your main process into two... It doesn't solve the problem, it just delays it...
I don't know how Go handles the situation where a process gets too many requests... Maybe it just drops requests (instead of adding latency like Node.js)? I don't know... But once a program maxes out on total available CPU, bad things will happen! It doesn't matter what language/engine is being used; whether requests will start lagging + timing out or that they will start to get rejected outright; programs cannot transcend CPU constraints.
I don't mind people switching to Go from Node.js because "they feel like it", but I don't think it's fair to write an article about it claiming that Node.js is unfit for technical reasons.
There is no meat behind this argument at all and I'm just tired of seeing it resurface again and again and again on HN. It's starting to look like a propaganda campaign.
Yes, but Go's concurrency model is far easier to reason about. "I _want_ to do this thing", and the runtime handles the _how_ and the doing. No need to draw up some IPC messaging protocol to handle trivial tasks safely. It's a language feature (ugh sounds so jingoistic). The cluster module is nothing like goroutines in concept or in practice.
You're making some mighty big assertions there, all the while disregarding the obvious weakness of a _single_threaded_ event-loop. I'm a total Node.JS fanboy since 0.2, but as was mentioned elsewhere in the comments, every language is making trade-offs: substituting shiny new concerns for it's own achilles heel.
>> I can't believe the CTO of digg bought this
>> Two weeks later, after my initial crash course introduction to Golang,
we had a brand new Octo service up and running.
>> you would also get 2x speedup.
Go is much better suited for large codebases than Node.js though, so when it becomes bloated again, it'll be easier to fix problems with/optimize the existing code and rewrites can be avoided.
Long flame graphs in node vs. golang is probably 50% refactor. The other 50% is the async calling pattern in node that leads to deep stack frames.
The overloaded event queue? If that is true the not the queue is overloaded but there are too many entries that are ready to run and at least one takes a lot of time running to completion. So there could be system level problems (flood incoming data) or a code problem. There could also be a low level i/o problem in node showing up under high load. It would have been interesting to learn the true root cause.
Considering the time already spent and the results they got and the cost of investment of switching to golang I would say the decision is rationale. The whole point of micro-services is to maintain the tooling freedom. You are trading off the need of expensive support contracts with the bounded risk of re-engineering a small service.
I was also thinking that Octo sounds way too much like a monolith. I wouldn't let retries tax the same instance.
I bet they could use something like RabbitMQ to sort work distribution and retries between instances without relying on nested callbacks.
To mention a big one, Go is mostly statically and strongly typed, in stark contrast to Javascript. Also it has significantly fewer misdesigns (such as the equality operator being weird). Are you claiming that such features don't matter for the effectivity of the platform?
I can appreciate that not everyone likes JavaScript - it's all a matter of personal preference I guess - but saying that Go is a better language because of the syntax is like saying bananas taste better because of their color.
They are two very different languages with each their strengths and weaknesses. Does JavaScript have a few design flaws? Absolutely! More so than many other languages? Quite possibly! Does it make me write worse software if I'm aware of those quirks? Most certainly not.
No but the difference between static typing and dynamic typing actually does. Go compiler performs a lot of optimizations that no Javascript engine can.
No need to bring up syntax. It's a superficial feature and has nothing to do with the actually important features I listed.
> Does it make me write worse software if I'm aware of those quirks? Most certainly not.
Is there some limit where good language features stop mattering? I guess we would obviously agree that creating web software with x86 assembler wouldn't be a good idea -- even if we could imagine a comparable library/framework situation.
If you are talking about the callback queue getting too full and causing delays you may have a point, but the objections that you have raised are quite petty.
Are you certain that that is true, or do you think it might just be that you're so used to the dynamic woes that you nearly ignore them?
That designing requires thinking, and thinking takes your attention away from whatever your goals are, whether you are used to it or not.
Strong type systems come with their own things that require extra thinking, but I think it is less than what is required to write robust dynamically typed code.
Many projects don't need truly robust code. Then, the dynamic language may still win.
Well, that's your bias. As I said in practice I don't see a problem. And also, as I said, there are static analysers, and you really HAVE to have strong type you can always use typescript.
But you seem to imply that you implement strong type checking into all(?) your functions yourself and think that is a better solution than having your platform do it for you automatically and systematically. Or is it about the freedom of not doing it?
Did the author tune the size of the libuv threadpool or did they leave it at the default of 4 threads?
Also, the response times given both for Node and their new system in Go are really not that great. The results for Go are an anti-climax considering it was a rewrite. They could probably undercut both times with a better design, regardless of language.
Node is a great control plane, and you can always use C++ as your data plane.
200ms of latency is quite normal for S3, not designed with that in mind.
Original thread based on Medium post.
Some languages make this workflow more obvious, and some people get stuck in the general way of thinking about problems. I think this is more of an engineering problem and thinking out of the box and ultimately his direction in looking for resources & help within the general community.
I think when you run into problems like these, you need to try to contact the community at large to ask about scaling, and ways scaling is generally done with these types of tasks. Thinking you're the only one who ever experiences trouble scaling is very juvenile and expresses inexperience & ultimately ego. Yes, he solved a problem, but he felt he had to uproot himself in the code already there to do it.
I guess he solved his problem, but he had a quicker path very possibly, he just didn't know it existed.
Are there Rails-like frameworks for Go? Why I try to Google that, I find lots of stories about people switching from Rails to Go, but not a lot of frameworks discussions.
Here's a good list of the major frameworks for Go. https://github.com/avelino/awesome-go#web-frameworks
Also, the complete Awesome Go list is a wonderful resource for browsing some of the best Go libraries out there.
I switched from ruby (and rails) to golang this year as my main language.
If you're thinking "I'm hesitating between ruby on rails and golang", you should probably go with ruby on rails (no pun intended).
Golang is more low level, and it expects you prefer conceptual simplicity over abstraction [EDIT: this is why you won't see prominent golang rails-like framework]. This also means you're supposed to already know how to architecture a web application (if that's what you want to do) and to know the techs behind it.
You mention you started to learn RoR, but do you have previous and consequent experience with web development? If not, definitely use RoR, it will provide clear guidelines about how to build a web app.
I think I'm going to keep plugging away at Rails. If I ever reach a point where scale is becoming an issue, then I've got a great problem to have.
The definitive answer is no. It's no because given Go's type system it is impossible to create a framework like Rails while staying type safe. One could create something that involves a lot of reflection tricks but you'd loose type safety. The alternative would be code generation + DSL but why use Go at first place if you end up relying on a complex DSL that needs to be compiled to Go first ? it doesn't make sense.
The best you'll get with Go are Sinatra or Flask clones. But Rails? no way.
If you want the Rails experience use Rails. Go is more verbose and more rigid than Ruby anyway.
Whether the author knows it or not.
Hence, this does not describe a good argument for switching to Go. It describes a good argument for better education, training and mentoring of developers so that they become programmers rather than "random assemblers of bits and pieces".
They could probably have solved this by scaling out, but I'm not going to fault their diagnosis or response.
Don't spawn off heavy requests without a throttle / queue, or else you will overload your single core.
We normally don't think about this because most requests are just tiny queries / small http requests.
I think a lot of people forget this sometimes and just spawn off unlimited multiple simultaneous requests in Node apps. Doesnt matter what language you're using, that's never going to turn out well
Just want to add that EC2 instances aren't exactly cheap either when it comes down to it. I'm sure budgets are tight at Digg. Far better to first try to solve the problem with refactoring than throwing more hardware at it.
I don't see it as a criticism Node.js either, it's just that Node wasn't the right tool for the job, in this case. If you pick the correct tools and know the limitations of your tools, then it becomes less important how proficient you are with those tools (not irrelevant of cause).
If you know the limitations of your tools then you can work around them. This was clearly not done in this case.
I have had node do 12000 requests a minute with no problem (on a single core). In this case they had 4 two core boxes, so they needed each one to at least perform 42 requests a second under max load.
I want to know more :)
This is a valid comment considering that the EC2 instance mentioned is a 2 core box, and there is a possibly that the node implementation didn't take advantage of both cores.
- Go has multi-core support out of the box
- Node.js has to be shown a little love, and has limitations. But this would have been a great use-case for Cluster.
If the Node implementation was using all cores, then I wouldn't have asked this question. As it stands I don't really know what we are comparing even with all the graphs.
Also we are still missing a little info on how they handled the requests. When you have a endpoint on a node server that is not a simple one, it is best to queue those before handling them (yes a additional queue). This prevents the process from choking because it just opened up NNN requests to S3.
Most developers are not working with hundreds of HEAVY requests a minute, but in Node.js this usually has to be handled with care, and does not work out of the box.
But if you want to pick up a shinier hammer then Go for it.
Personal opinion: Because comments like your original one are annoying as hell.
They never fail to show up. Someone post an interesting article one how they solved a particular problem and someone always have to show up with these pointless comments. Not everyone will know about, in this case Cluster, and it's not relevant without an explanation.
Article and blog posts about Go seems to attract comments like: You might like Rust/Elm/Elixir and "that problem could have been solved using X/Y/Z package or framework".
More often than not it's missing the point, which often is people, with little or no Go experience are managing to solve really complex problems within very short time frames.