From a Ruby monolith to microservices in Go
sourcegraph.com
sourcegraph.com
For example, if you're in the situation where you've gone from 12GB to 5MB of memory consumption, then you have a design issue and not a language one. That's not Ruby's fault.
We are also in the process of converting several monoliths into a microservice-driven system, but we are doing so with Ruby. It remains great language to use for building web apps, and I suspect some of the push back against it is caused by a dislike of the cliché giant Rails monolith that we've all come across, with hundreds of gem dependencies and so on. Rubby and Rails have a habit of encouraging such bad designs primarily through ease of use, but it's hard to see that as a problem with Ruby.
Go is a good tool for doing the same, and single-binary deployment is pretty magic. I'm sure it'll be supported for ages, so if you want to re-tool to use that then go for it! But Ruby's a viable option too, and it's important not to place blame for bad app design in the wrong place.
while it's true that there are plenty of projects and services that reduce the pain of that, I'm certain that everyone who's ever built a Ruby app has experienced deployment hell at least once.
I thought the same thing, I find that figure also very hard to believe. Perhaps I just lack experience, but if you get 12G of memory usage that drops to 5M after rewriting it (whether another language or not) seems to me as if they did something pretty wrong in their previous iteration.
The X is largely dependent on how many dependencies your app pulls in, on your garbage collection settings, and on how much state you carry around with you. I don't know what an off-the-shelf Rails app requires as a minimum, but as a comparable, BCC requires ~120 MB per process and AR requires approximately ~200 MB. Neither are particularly mammoth code bases.
Thus, if AR develops a business requirement for 100 requests per second on something which takes 200 ms per request, my server budget just increased by 4 GB of RAM. That is absolutely unavoidable in the Rails concurrency model. It doesn't matter how well you code the small thing at issue: you pay a memory tax based on desired maximum volume and app complexity, always.
(Neither here nor there, but AR has features quite similar to the notification feature that the app described in this article. We handle this by using queue workers -- which each require 200 MB, as described above -- and the decision that queues do not require immediate servicing. Instead, if thousands of calls are dumped into the system, we add them to a queue quickly and then consume the queue leisurely. Different apps are different apps, but I'd wager a guess that we do substantially more volume in terms of calls on ~400 MB of RAM.)
Also neither here nor there, but engineers are expensive and gigabytes are cheap. There exist many circumstances in which the architecture which requires 12 GB of RAM is a defensible engineering tradeoff versus the one which takes 5 MB. As Thomas told me once when I was asking him how to cut 100 MB off of Redis so that I could avoid upgrading a VPS by 1 GB: "How much does the extra gig cost? $10? Why are we even having this discussion?" And he was right.
Edit: Our requirement was to process this queue as fast as possible and that means more workers. With process based concurrency that is very costly as you have explained.
Btw how are you generating the PDF from HTML and are able split a single HTML into multiple PDFs?
And I'm not splitting pdf but splitting html generation work load, and then create individual pdfs from those html chunks. Then they will be joined together (using pdfunite). I found this much faster then joining html and generating large pdf.
(I'm planning to test with PhantomJS 2)
Also, if the workers shared all the code, you could have used unicorn to fork the processes after the code loading was complete. The 400MB per process would then instantly come down to something ~10MB per process at which point rewriting would have been delayed for another year or so.
Unicorn forking benefit is overrated, we used it and we don't see much benefit for long running processes.
Sidekiq is good alternative but that means some rewrite(for our app anyway). Secondly Sidekiq looks mature today, I started working on some of these changes 2 years ago.
2 years ago, ruby was not COW friendly. So yeah, there was not much benefit to forking if you were using 1.9.3. Not sure how well does ruby 2.x fares in that respect
So, in any case you have to rewrite as much as code that I rewrote in Go and decoupled from main App. (its not lot of code, I mentioned in talk)
This is a vital point — everybody wants to shoot for lovely, elegant systems that use minimal resources. But resources are astoundingly cheap; we've got a bunch of servers with 128GB of RAM, and you can get these for around £100 a month each. That's less than two hours of developer time. If using Ruby in place of another language saves you that much time—or even just makes you a little happier—it's probably worth it.
"This is a function of how Rails does concurrency, which is by using multiple processes."
Multi-process vs multi-threaded is an application server architecture matter, not a Rails matter. Yes, your application must be thread safe, but this has been available in some form for Rails apps since a long time now [1]. The biggest problem in recent history has been Gems and their thread-safetyness (is that a word). The number of threaded Ruby application servers has been on the rise for a long while now, and even Passenger (traditionally the multi-process front-runner) is going threaded in version 4.0.
1: An interesting blurg post related to Config.threadsafe! (which was a huge source of confusion): http://tenderlovemaking.com/2012/06/18/removing-config-threa...
But Paul Graham knows exactly why you are sticking to the past (just replace 'Blub' by 'Ruby'):
As long as our hypothetical Blub programmer is looking down the power continuum, he knows he's looking down. Languages less powerful than Blub are obviously less powerful, because they're missing some feature he's used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn't realize he's looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.
When we switch to the point of view of a programmer using any of the languages higher up the power continuum, however, we find that he in turn looks down upon Blub. How can you get anything done in Blub? It doesn't even have y.
By induction, the only programmers in a position to see all the differences in power between the various languages are those who understand the most powerful one. (This is probably what Eric Raymond meant about Lisp making you a better programmer.) You can't trust the opinions of the others, because of the Blub paradox: they're satisfied with whatever language they happen to use, because it dictates the way they think about programs.
The point remains though; it's not that the poster doesn't know how to build a 'proper' system in Ruby, but that the frequent articles we see with the theme "we rewrote our system from language X to language Y and it's much better!" are rarely helpful, because (I can't stress this enough) architectural design is vastly more important than what language you are using to implement your system.
This leads to a dangerous bandwagon-jumping faddism where developers start jumping on to the next big thing, because they assume it will solve their problems. We saw exactly that with Ruby, for example; developers assumed that they could escape from the verbosity and enterpriseness of Java just by changing language, ignoring the pitfalls that could be experienced.
PG's 'blub' example is a very useful allegory, but it doesn't apply here. I'm not at all suggesting that Ruby is the perfect solution to all problems, or that using Go is wrong – I use both of them! Just that saying 'Go is better than Ruby for writing web apps because we reimplemented everything and it was faster' is not helpful.
We saw exactly that with Ruby, for example; developers assumed that they could escape from the verbosity and enterpriseness of Java just by changing language, ignoring the pitfalls that could be experienced.
this worked for a lot of people until they got to the architectural stuff. but going from Java to Ruby (infinitely nicer) is distinct from going to J2EE to Rails (easier to get started, harder to keep going).
So here we've got a Ruby/Blub programmer looking down the power continuum at Go and sees language deficiencies. He looks up the power continuum and sees maybe Haskell and Clojure etc. Since Go was created[`1] to be down the power continuum and contains nothing weird, new, or even particularly different, I don't think the adage holds.
[1] citation needed
Seriously, why do you think Go was created up the power continuum? It may be, but what features or idioms (barring channels) put it "up there?"
That is one dangerous assumption you made there mr. Performance (in terms of response times, IO waits, etc) can be severely affected in distributed designs.
That's why there's an 'almost' in there. You're right that distributed systems have a different set of challenges, but if you're developing a system that requires the kind of incredibly fast response times that will stress a distributed system, you're already into territory where general 'this is good guidance for web apps' advice doesn't apply to you.
The OP made his post as objective as possible, politely written and nobody is loosing his face, neither Ruby as a language nor the Ruby community. But people are still resentful and fire back.
I believe it's not a discussion about Ruby anymore, its strengths, its weaknesses and wether Ruby still fits in 2015—no it's the fear of people that their core competence might be less worth in future. That all the time and energy they invested all the years into one language might be not a good investment anymore. The fear that better tech will replace their tech and that they have to start at zero again.
I know I am going to be heavily downvoted for this post. Before you downvote just think a second again why you'll downvote me.
The point is that articles about rebuilding entire systems in different languages conflate two things — an architecture change, and a language changes.
This often results in the sort of appraisal you see here – 'Go is better for us than Ruby, because we completely rebuilt a system and it's now better'. That's got some value, but it's frustrating to see it used as something of an absolute metric regarding the suitability of a language when it's conflated by the far more important architectural changes.
So I think your view that there are a bunch of developers who are afraid of the future is a little shallow; rather I suspect there are a bunch of developers who saw exactly the same sort of thing happen when Ruby was becoming popular, and realise that extracting helpful data when there's an ongoing fad ('we're jumping from X to Go') is rather difficult.
If you replace every mention of Ruby with Java, and every mention of Go with Ruby in the comments on HN, then you've invented a time machine to go back 10 years.
And if you do this with C, welcome to 1995.
They could have rewrote it in java in a similar amount of time, and with the same results. And they could hire from the large pool of expert java devs. Of course that's not "cool".
Each java based microservice requires its own packaging of jvm, jars, etc. Without this it is not easy to replace individual microservices, as the cross dependencies between the microservices will take away a big advantage of going that route.
Go provides this level of deployment separation for free by default.
1) Ruby is a complex language: Ruby is quite simple. Everything is an object, every method call is a message send that follows the exact same protocol of dispatch. Also, completely irrelevant to the main thrust of the article.
2) Memory consumption and speed: Can't really refute anything since by their own admission they were running a monolithic system.
3) Concurrency: Plenty of options. I like celluloid personally and if you switch to JRuby then you can do all sorts of other stuff as well with automatically managed thread pools.
4) Backcompat: Ruby is not an enterprise ecosystem. Some people call that speed and agility. In their case they really should have gone with Java since they are basically serving an enterprise customer. This is not a knock on Java. I think the language and its ecosystem is by design made for stuff like this.
5) PDF generation: I'm guessing the Ruby code was calling some C library to do the rendering and they swapped out all of it for Go. I don't know what library Go is using to render PDFs if it is the same library then there is something interesting happening here. If it is a different library then they are comparing apples and oranges and there's nothing to see here.
6) Alert service: Completely bound by I/O so doesn't matter what language you use. Bad design for keeping many processes around because you only need to spin them up during spikey times which can be easily handled with better engineering. Also eventmachine and celluloid for the worker internals would have been another option so again they didn't do their research.
7) Gradebook service: No idea again. Thin on details so nothing to refute really.
8) Deployment: Sure. Why not. Single binary is easier or JRuby and a single jar which is just as easy.
In conclusion, they re-wrote everything and I'm sure removed a lot of cruft in the process and called it a victory for Go. In a year or so they're going to run into similar issues and Go is not going to save them because from what I read all I saw was bad engineers blaming their tools.
I would exercise a bit of humility when commenting about other people's architectural decisions: they have bled all over that codebase and infrastructure, but I have not. They see it in their dreams, and I do not. They may from time to time decide to tell the community about their adventures. That is an excellent opportunity for me to learn from another engineering team's perspective and experiences, rather than an opportunity for me to demonstrate my intellectual superiority over them. They do not, as a function of having written about their experiences, owe me responses to my off-the-cuff analysis of their engineering tradeoffs.
Once again, I'm not aggressive but I am critical. nexneo has since addressed all the issues people have brought up and if he'd done that initially then I would have been much less critical.
At the beginning, a Monolith has huge advantages: easy to deploy, all in one place, smaller operational overhead, easier to manage with unclear requirements.
Later, it's easier to find out what you can split out and you can learn how to route and manage things piece by piece.
Only risk is, you break things into many component then you should so balance is required.
My point being: I usually rate architectural changes as more important as a change in development details (and the programming language might be a big one there, but it still is one, IMHO).
I'll tell you why: a lot of the sentiments people bring when they now switch from Ruby to Go, I've heard before. When people started Ruby - I was already doing Ruby for ~2 years before Rails even came out at it got me into the position of saying: "just you wait until you see the bad parts". They attributed a lot of things to Ruby, while they were really changing their development model.
The same number of workers (that required 12 GB in Ruby) now requires just 5 MB of memory.
Not that I don't approve of pro-Go articles, but doesn't this maybe indicate that something was wrong or inefficient in the Ruby implementation?
Our takeway was that the (Rails) monolith model tends to put a lot of things into one process space. Controlling library loads can be a chore.
I could have rebuilt in Ruby and that was my first thought but deploying and maintaining Ruby apps are lot harder then you can imagine. I didn't like idea of maintaining lots of small ruby apps. Once you deploy single Go application, you might not want to deploy another Ruby app.
We do a lot of non-Rails Ruby-work and did implement Microservice-Architectures in it and found it rather unproblematic. The trick is not to start with a full-stack and build down, but to start with a small kernel of an app, deploy it, add something, deploy it, iterate, iterate, iterate.
I don't use the Ruby stack, but if forking happens after loading libraries, this is not true, since most UNIXes (such as Linux) use COW memory pages when forking. So, it may appear that you use N times the memory of a single process, but most of their memory pages are shared.
See fork(2).
Only in recent Ruby versions is that not a problem (and was changed for exactly that reason).
But yes, your memory report might be read wrong.
I haven't done a thorough or systematic comparison, but I'd expect slightly slimmer but similar results with Sinatra apps given the 80MB figures you quote above, the rails processes were around 120MB on average, Go 6-10MB and scales without extra processes. Obviously the smaller the framework you load for each process, the less memory you're going to use, but there are also architectural differences in the languages which seem to add up to significant memory usage in the case of Ruby.
Because to scale ruby you'll typically add more processes, and to scale go you'll just let it start another thread (the http server does this for you), that difference adds up to a significant difference when you have a lot of traffic.
That's not to say that Ruby is impossible to scale and I love Ruby the language, but it is a bit slower and a bit more memory intensive than Go.
Also, you can move to JRuby if you want a one-process/threaded model, which will get memory-conservative very quick. A JRuby thread will quickly get you into the ranges that you quote for go + the base cost of the JVM.
Point being that memory comparison even in the Ruby/Rails space are incredibly problematic, often influenced by the runtime (and heavily so). Still, the discussion is incredibly simplistic (for example by assuming that you are running unicorn/fork).
If a discussion about memory issues does not include an exact reference to the runtime and implementation of the system used, I would consider it void immediately.
Same for Go.
Rails is opinionated, yes, but I'm generally pretty satisfied with their opinions, and it's nice to have a full Batteries-included package.
As far as application structure, I'm not really sure what you mean. Go projects are divided into packages which can be any number of files. At some point you'll have a main package which has a main() function which gets called to start your server. It's similar to C, for example.
That said, the variety can also be a plus. Some people chose to use something like Martini that does dependency injection. Others think that's not a great idea.
Later I started use just standard library with http://www.gorillatoolkit.org and used fswatch for hot reload. Now there are lot of alternative in routers but no clear winner.(and thats good thing) I will suggest go with standard library and simple router but be ready to revert back to framework like revel if that doesn't workout.
At one place "form-urlencoded" is also used, so its not only JSON.
Does json-rpc have a standardized crypto wrapper, though? That seems like the most important reason to use https. Although json-rpc + nacl/sodium might be better in theory...
JSON has advantage of human readability and for that tooling support is required which probably you will loose with RPC. For example I use many tools to inspect requests that won't support it with RPC
It looks like they do a good job of tackling performance problems as they see them, even at the architectural level. My guess is they didn't tackle this because it wasn't a problem worth prioritizing compared to the compounded shitshow of a monolithic Rails app.
Don't underestimate the importance of toolchain support. Want to spin up a Ruby microservice which speaks HTTP? Sinatra and you're done. Go? Whatever the Go HTTP library is and you're done. Need to interact with it from the command line? Curl and you're done. How about from an automated testing script written in Ruby? Net:HTTP/HTTParty and you're done. Thinking about how to deploy it vis-a-vis firewall/etc? Port 80 and you're done. Need coarse-grained access logging? Built into Nginx/etc already; you're done. Uptime monitoring? Expose a /monitor endpoint; provide URL to existing monitoring solution; you're done. Deployment orchestration? Use Capistrano/shell scripts/whatever you already use for the app proper and you're done. Encrypted transport? HTTPS and you're done. Auth/auth? A few options that you're very well-acquainted with and are known to be consumable on both ends of the service trivially.
Edit to note: I'm assuming, in the above, that one is already sold on the benefits of a microservice architecture for one's particular app/team and is deciding on transport layer for the architecture. FWIW, I run ~4 distinct applications, and most of them are comparatively large monolithic Rails apps. My company's bookkeeping runs in the same memory space as bingo card PDF generation.
Things that would tilt me more towards microservices include a very rapid deployment pace, large engineering organizations which multiple distinct teams which each want to be able to control deploys/architecture decisions, particular hotspots in the application which just don't play well with one's main technology stack (as apparently happened in the app featured in this article), etc.
For a finer-grained application, such as the way Facebook generates pages from about 20 servers feeding data in to be assembled, that wouldn't scale. Facebook has Thrift and Google has protocol buffers. Both work, but result in a somewhat complex build procedure with a pre-compiler for interface specs.
(Serializing and de-serializing data into and out of databases and networks is either interpreted and slow, or compiled through clunky tools. Technology in this important area still sucks. There's a long history of clunky solutions to this problem, from DCOM, CORBA, and ASN.1, to SOAP to REST/JSON.
Part of the problem is political. If you expose an API defined in any of those except REST/JSON, there's a formal, checkable definition of the API in machine readable format. If the API doesn't match the spec, the server is broken. With a REST/JSON API, you get to blame the caller for doing it wrong. This is convenient for API developers.)
PDFs are one conceptual bucket that can have their own datastore and a simple API. Sending email isn't included in that service's contract, as it actually outsources this task to Mailgun.
REST/JSON can have formal specification – check out http://json-schema.org/. Swagger (http://swagger.io/) is another example of this and works really well for automatically generating client libraries in many languages and auto-updating documentation.
Why do people keep getting amazed about the performance they get when using native code in their applications instead of a naive interpreter?
Not bashing Go, the title could be "From a Ruby monolith to microservices in <pick your language with native compiler available>".
Hey, this could probably even be possible in Ruby, if something like RubyMotion was generally available.
I did in Go not because of language is better, ecosystem and culture is better but not language part. I will suggest you to do small real life project in Go.
Thanks for the suggestion, maybe you should check my history.
RubyMotion as an AOT compiler would bring many of the same benefits in terms of performance, while staying in Ruby.
Apparently only Lisp, Scheme and Dylan enjoy the existence of AOT compilers in their toolchains.
What I read this to mean is that the Ruby app accesses the microservices over HTTP with some kind of token-based authentication. This would make sense if there some form of shared session store which each microservice could validate the token against. What I'm confused about is how nginx fits into this? Is it routing requests directly from the client to microservices? Or is it a proxy layer in between the Ruby app server and the microservices which allows the Ruby app to forward auth-token information? In the former scenario, would the Ruby app server be pinging the same nginx instance that forwards it requests in the first place?
You might see significant advantages moving to ruby microservices if you have high traffic and a complex monolithic app which would benefit from being split up. They are not a panacea of course as they introduce significant complexity, whatever language they are implemented in - it's a matter of trade-offs and once your monolith reaches a certain size it might be worth splitting it up.
Re libraries, most of the ones you mention have analogues in go or are pretty simple to replace, save AWS, which is apparently coming soon:
AWS - https://aws.amazon.com/blogs/aws/coming-soon-aws-sdk-for-go/
Sidekiq - go myFunc()
Devise - bcrypt.CompareHashAndPassword + the mailer functions
Deployment - Rsync or Ansible - this is simpler in go as all you require are your executable + templates
Time taken: 1-2 days.
Their high memory consumption was caused by large numbers of unneeded dependencies (e.g. Rails) included in their alerting components. By refactoring so drastically into a separate Go service, they may have squandered tens of thousands of dollars of company/investor money in retooling, retraining and learning a new environment. Good for the programmers, toxic for the company.
I don't think anyone disagrees that refactoring and evolving the code would have resulted in gains even if they stayed in Ruby. I'm sure most of us would even agree that if you wrote something, then immediately scrapped it and started over, you might already be able to improve on your original based on some new knowledge you've gained from the experience.
When a company does this, they not only learn new things about the language they are converting to, but they are gaining new and valuable insight into the language they already have used. Posts like this are valuable to all of us because they are sharing their experiences; whether you agree with their choices or not.
85% of our server codebase is still Ruby.