Thanks, you explained it very well.
Edit: Our requirement was to process this queue as fast as possible and that means more workers. With process based concurrency that is very costly as you have explained.
Edit: Our requirement was to process this queue as fast as possible and that means more workers. With process based concurrency that is very costly as you have explained.
Also, if the workers shared all the code, you could have used unicorn to fork the processes after the code loading was complete. The 400MB per process would then instantly come down to something ~10MB per process at which point rewriting would have been delayed for another year or so.
Unicorn forking benefit is overrated, we used it and we don't see much benefit for long running processes.
Sidekiq is good alternative but that means some rewrite(for our app anyway). Secondly Sidekiq looks mature today, I started working on some of these changes 2 years ago.
2 years ago, ruby was not COW friendly. So yeah, there was not much benefit to forking if you were using 1.9.3. Not sure how well does ruby 2.x fares in that respect
So, in any case you have to rewrite as much as code that I rewrote in Go and decoupled from main App. (its not lot of code, I mentioned in talk)
Btw how are you generating the PDF from HTML and are able split a single HTML into multiple PDFs?
And I'm not splitting pdf but splitting html generation work load, and then create individual pdfs from those html chunks. Then they will be joined together (using pdfunite). I found this much faster then joining html and generating large pdf.
(I'm planning to test with PhantomJS 2)