A Private PaaS with Mongrel2 and ZeroMQ | Hacker News Reader

A Private PaaS with Mongrel2 and ZeroMQ

notes.ceondo.com

77 pointsthomasriche24 comments

I am the author of this PaaS, I wanted to push this note on HN Monday to have time to answer the questions, but the cat escaped the box faster than I wanted. I am here to answer questions but do not expect "instant" answers.

I built a very similar PaaS with mongrel2 about a year ago (though all applications were untrusted and running inside an OpenVZ container) - only to change it to be nginx/HTTP-based because of stability.

Great to see this technology is ready nowadays and can be used for production systems :)

Mongrel2 is still beta, but with a good process supervisor, it still way more robust than a RoR setup from the early days. Once in a while, it can die, I still need to instrument a bit more to track the conditions where this is the case.

On the other hand, ZMQ for the backend is just a pure pleasure and I haven't had a single failure of my components for more than 1 million real requests involving many ZMQ backend interactions.

Interesting, my problems back then were actually ZMQ-related.

Though to be fair, we were making it a bit complicated by suspending/resuming the OpenVZ containers. From that we sometimes got dead ZMQ connections that were black holing all traffic (incredibly hard to debug - ended up switching because of it).

OpenVZ is tricky, they are patching the kernel pretty deep and this creates a lot of hard to debug issues. I had a lot of troubles with containers running software mapping big files like MongoDB.

Yes, most of my crashes on the mongrel2 server I run come from ZeroMQ too. They fix them when I find them, but I actually run my servers under valgrind continually so I can catch them and report them.

Very nice post, thanks! I'm looking at experimenting with Mongrel2 to see if it would be a nice fit at work, where we still use apache2/mod_wsgi (we have modest needs) to interface our Python + CouchDB app. However, we've started adding WebSockets stuff to the system, and it would be nice to run everything in a slightly more coherent matter. We also use ZeroMQ to pump messages around the network (that later get converted into WebSockets frames for web frontend consumption).

- Is mongrel2 getting good development? The latest release was in June, and the latest commit in the canonical repo seems to be from July...

- Is there good support for RFC-version WebSockets yet?

- Is there easy/performant stuff for simple asset serving via mongrel2, or should I use nginx for that?

1. Zed is coding in bursts, check the develop branch, it is pretty active and a 1.8 is around the corner.

2. The same, in the develop branch, there are still some questions on how to best implement it like which parts must be done by the handlers and which by Mongrel2, but the coders involved are good and very pragmatic, I bet the results will be a high quality implementation.

3. http://notes.ceondo.com is served by the "Dir" handler, that is, direct static files from disc without going through the asynchronous ZMQ based sequence. It is pretty fast.

Of note: There are several of us in #mongrel2 on Freenode, and there are periodic patches from us :).

Hey, just a quick reply, that, yes I'm rather busy right now but I still squeeze in time for Mongrel2 and should be getting out a 1.8 beta soon.

I've been working on the WebSockets support. I changed the handler <-> mongrel2 interface for websockets a bit, and I think I've reached the right point in the websockets-devel branch. If you are interested in websockets you can check out that branch and see what you think. If I get no negative feedback I'm merging it into the develop branch once I've tested it more

Interesting username. Kind of reminds of me something... oh yeah, low orbit ion cannon, a DOS tool I have battled in the past.

Looking forward to your explanation.

It's also a popular French name (I know 2 Loics personally...) No idea why this particular user uses this username but, it might not be for the reasons you think...

My name is Loïc d'Anterroches. This relatively old, from disputed origin, French forename is not widely used. I can understand your reaction if reading my username brought memories of fighting against a DOS attack.

Wow, OK, sorry. I had no idea. Thanks for educating me.

I'm looking forward to your explanation of how you're posting from 10 AUs away from Earth.

This is a great article exploring what some technology behind PaaS might do, and also a great start, but falls short of true PaaS functionality.

* Many PaaS offerings (like AppFog) are not like shared hosting at all, only the free offering hosts people multi-tenant, when you pay you get dedicated VMs

* The platform in platform-as-a-service means it connects you to fully managed services (on Heroku there are all the add-ons as well as Postgres, on AppFog there are add-ons and MySQL).

* One of the most compelling parts of PaaS is that it is managed by someone else so that you don't get a call at 3am when it is broken

Moreover, if you were to run a PaaS on your own, it would make more sense to try Cloud Foundry which has specifically addressed at least the services aspect of the above comments.

I think you missed the "private" and you jumped a bit fast.

1. Everything is running on dedicated hardware.

2. The system provides MongoDB, PostgreSQL too, it is just managed by us because we need this level of control to put PostgreSQL on SSD and special high CPU instance for scientific computations. Cf point 1.

3. Hosting services for the past 10 years, crossing my fingers, I have never been called in the middle of the night. But yes, the risk is there.

For Cloud Foundry, I don't know it, our solution is extremely light on resources we have full control of it and we are extremely happy with. We can hack/extend as we want and all in all, it is very few lines of code and I personally love simplicity.

Update: Ok, you are from a "Fog" provider, now I understand the defensive tone of your post. Do not worry, the goal of this note is to share what we do and give people willing to go this way some inspiration. Not trying to compete with you :)

I was mainly responding to the "Connecting the Dots, What is a PaaS?" section which I thought was the weakest part of your article. Otherwise well done.

Ok, I understand your point of view. But if you look at the current high quality shared hosting offers, you have managed dedicated MySQL instances included and many of such add-ons.

Anyway, we created this PaaS because it brings us real benefits and a competitive advantage. For our case (not the usual one as we need to run scientific computation with specific hardware and software) it is better to have full control of our hardware. We are outliers and most of the people would be better going with a good PaaS provider.

Thank you for this - it's a great example of how to do it properly and on how to choose the right tools for the job.

Unfortunately the organisation I currently work for managed to screw it royally. Firstly, we mandated .Net as the technology paltform (which is the most unscriptable and obtuse platform ever in the history of time). Secondly, we made the deployment too complicated (one deployment is 30+ VMs and about 20 SQL server databases!). Thirdly, we wrote our own tooling entirely (there is a lot of Not Invented Here in the .Net space). Fourthly, we let management in on the process (who don't know arse from elbow).

Hey now. There's probably a bit more blame on .Net than there needs to be :) I agree though, there seems to be quite a few wannabe .Net architects who design unmaintainable behemoth deployments just because they can. In fact I'm helping one company completely replace their old system because all the developers who built it left in a hurry and no one now knows how to run the custom deployment. But that's not the language's fault. There really are some good companies out there doing awesome things with .Net

I went through some of your links but didn't find the whole shebang. Is there a plan to release all of it.

I love the detail that we went through when describing it.

It will be released in parts, the core components are already available:

- distributed process manager: http://projects.ceondo.com/p/diprocd/

- the pre-receive hook taking care of packaging and deployment: http://cdn.ceondo.com/bareku/buildpack-python.tar.gz

If you need something similar, you will most likely adapt it heavily for your own setup, so, I am not sure if a full release is feasible.

Reply on news.ycombinator.com