Rendering 16,000 Schematics in the Cloud with RabbitMQ and PhantomJS | Hacker News Reader

Rendering 16,000 Schematics in the Cloud with RabbitMQ and PhantomJS

circuitlab.com

35 pointscompumike20 comments

Work queues are a mostly solved problem, unless you've got unusual needs, like preposterous amounts of throughput. These guys don't have unusual needs, and they've got a paltry 16k tasks to keep track of. For that kind of job, use a library. You don't want to be the one who writes code to deal with restarting hung workers, or shunting pathological jobs to a retry-later queue, or changing pipeline topology live in production without a hiccup. I've written this stuff -- we had unusual needs -- so take it from me: you don't want to write this stuff.

If you do most of your work in Python, Celery is pretty slick: http://celeryproject.org/

If you like Redis -- and who doesn't? -- then Resque or one of the alternate implementations will work well in Ruby or a plethora of other languages:

https://github.com/defunkt/resque

https://github.com/defunkt/resque/wiki/Alternate-Implementat...

Beanstalkd is pretty generic, and simple to set up. If for some reason you can't use Celery or Resque, maybe have a look at it:

http://kr.github.com/beanstalkd/

Distributed task queueing is important and useful stuff, and if you half-ass it, your infrastructure will literally explode, usually at 3:00 AM. Get a good library for it.

Indeed, Celery was a tool we considered using for this task, especially since we're Python/Django based already. And yes, there are a lot of seemingly minor but quite important sticky points as you mentioned (and we listed in our post). I would certainly give these libraries a close look if I were to implement another work queue today. I'm in total agreement with the basic message: distributed task queueing is important and complex, so don't "half-ass it". :)

AMQP is overkill in most cases (I found out from personal experience). The OP has a good point in using Celery.

See my comment above if your rendering clients are written in Erlang.

Can you clarify what you mean by "use a library?"

It's easy to follow the rabbitmq tutorial and build a nice little pipeline for tasks - what is the difference between that and using a library?

Quite a lot if you don't want stuff to break.

The key question is, "What happens if $INSERT_CALAMITY_HERE?"

What happens if a worker crashes? What happens if a few jobs cause the workers to work forever? What happens if a job hits a bug in your worker code and causes it to reliably crash every time? What happens if your queueing server goes down? What happens if you need to migrate your queueing server to another EC2 instance because Amazon has decided to get rid of the physical machine it's running on?

Undesirable answers to such questions include "We accidentally drop a job without noticing," or "We have to shut down the whole system in a user-visible way for twenty minutes while we do maintenance," or "All the workers keep crashing and being restarted, flapping eternally until someone notices and manually intervenes."

EDIT: Oops, I just noticed that I didn't really answer your question about what I meant by using a library. What I mean is, use one of the more full-featured job queueing libraries like Celery or Resque, rather than trying to roll your own on top of something like RabbitMQ.

Ah, okay, so something like Celery would handle all of those kinds of cases for you. (Though... doesn't rabbitmq have many of those features built-in? Queue server replication and redundancy and stuff? Though I suppose a dedicated project would handle more of those calamities than I could think of on my own.)

Thanks for that!

RabbitMQ doesn't have all that built in at all. RabbitMQ is only an AMQP message broker/exchange/queue agent. It's hard to articulate (and probably should be done in a blog post) all the things that can go wrong, but rest assured, there are a lot of things that can go wrong with RabbitMQ AND with your worker nodes that are built on-top of RabbitMQ.

Distributed workloads is a hard problem, but it is a solved problem (for most applications, there are some exceptions thought too).

All in all, creating our professional-quality schematic render outputs in a variety of formats is a CPU-intensive process, typically requiring roughly 10+ seconds of a single core of modern CPU time per schematic.

Wait… really? A schematic editor can render a schematic at least 30 times per second. "Rendering" to PDF should be even faster since PDF is a vector format. Are they rendering in hundreds of different formats?

So you can load this from a schematic format, parse it, get all the element symbols loaded in the right places, grab and render the fonts and place them appropriately, and construct a well-formed output like https://www.nerdkits.com/media/circuitlab/20120620-democircu... , all in 1/30th of a core-second? With a bunch of disk seeks in there too?

Of course, we could devote engineering effort to bring that 10 seconds down by an order of magnitude, writing a fully custom PDF (and EPS, SVG, PNG, ...) rendering engine, but part of the beauty of elastic cloud computing is that we can make better use of existing components -- even if it's less efficient from a CPU time perspective -- because the CPU time is cheap and programmer time is expensive (and better used elsewhere). There's a balance, of course, but making well-informed tradeoffs is what engineering is all about.

Uh.. yes? Obviously you should have all the elements, symbols, fonts already cached in memory. Its not like you have thousands of these, anyway.

Layouting was already done by the user, so I don't see what part of rendering is incurring the heavy costs here. As it stands, you have now spent lots and lots of programmer time building a batch system from scratch.

Premature optimization can be an expensive mistake to make... It may feel "wrong" to use 10X or even 100X CPU cycles when you "know" that you could do it in X, but if the 100X cycle solution still fits the business requirements of the overall system, and takes a lot less programmer time, which path would you choose?

If, as a progammer, I see something like this render taking 10 seconds, I will shake my head in confusion and think "what the hell is taking up all this time". Stuff like loading the same font a hundred times, allocating memory for all the same symbols - identifying and getting rid of that is not premature optimization, its plain fixing the code.

Of course you might have done all of that - no way to know unless you tell us more about the actual pipeline.

But this sounds more like 'no optimisation', and a lot of complexity to maintain. I'm scratching my head over the toolchain chosen. The output is PNG, EPS, PDF, SVG, but you're using Inkscape and Ghostscript? Something seems off there. Cairo has all of those backends already, it's fast and you can call it from python (via pycairo). What's more, Inkscape already uses it as a backend. There's even another schematic editor (gEDA) that uses it, AFAIK.

But, I presume you didn't use cairo for a reason - or you are using it, but need those other tools to post-process its output somehow. I'd be interested to know why?

Premature optimization is not about "optimizing before you need it", it's about optimizing before you've measured it. In this case, you have already measured it; you know the cost. There's nothing premature of optimizing at this moment. I would consider spending a day profiling the current setup to figure out exactly how much you could optimize.

> and takes a lot less programmer time

I don't see how this is true. You've already spent time setting up a distributed system from scratch (pretty much at least), and you also need to maintain it. Distributed systems are complicated stuff; lots of stuff that can go wrong. If the optimization made it possible to use a single machine, it would be much easier to grasp and debug.

writing a fully custom PDF (and EPS, SVG, PNG, ...) rendering engine

You mean like this? http://cairographics.org/ (Which, BTW, has Python bindings…)

because the CPU time is cheap and programmer time is expensive (and better used elsewhere)

Yes, but programmer time to implement rendering using a fast library such as Cairo is O(1), while the CPU time is O(n) where n is the amount of business you do.

(Disclaimer: I have written a schematic renderer before using Cairo. It took me maybe a couple days.)

What does having Erlang experience provide for you when using RabbitMQ? Unless your rendering nodes were written in Erlang... In which case you could have easily just used Erlang's built in node connection primitives and used RPC's instead of the much more complicated AMQP process.

I know this from personal experience. If you have a consumer based distributed worker pool, AMQP is overkill! My application is built in Erlang and the worker nodes are also built in Erlang - I initially went with using RabbitMQ to hand work out to the workers (and to receive processed results back from workers) but I quickly found it added a lot of message latency (vs. a straight RPC call) and I also had to write a lot of management code.

Switching over to straight RPC was a great decision and I essentially kept the "worker queue" logic in our Python application (where it is a "solved" problem).

Big topic, for sure - I should write a blog post on how my application is structured.

Wow... CircuitLab is like a modern, web-based Spice. I love this idea. I will be using your site to quickly generate images for papers and presentations, and I will certainly recommend it to students. Awesome!

(Are you using Spice as the underlying simulator?)

Thanks! Please do pass it around :)

We actually have a custom simulator engine running client-side in Javascript.

Nice! :D

Reply on news.ycombinator.com