Building Clojure services at scale | Hacker News Reader

Building Clojure services at scale

blog.josephwilk.net

137 pointserkose21 comments

I have not worked at the scale discussed by Joseph Wilk, but I am pleased so far with my ability to build out micro-services with Clojure. The eco-system around Clojure seems to get stronger every day. I'll mention a few of the libraries that I'm using.

For building out a RESTful API, I found Liberator to be very clever and original:

http://clojure-liberator.github.io/liberator/

For handling internal queues, I use Lamina:

https://github.com/ztellman/lamina

For instance, when saving something to the database, I usually put the data in a closure, put the closure in a Lamina "channel" and let some workers execute the closure when the thread-pool and IO constraints allow.

I just started experimenting with the Ribol conditional restart system, for error handling:

http://docs.caudate.me/ribol/

For the integration of frontend with backend (MongoDB) we were originally going to use DerbyJS, but I talked the team into trying Sente instead:

https://github.com/ptaoussanis/sente

We are not done, so the jury is still out, but I am very impressed, so far, with Sente.

For dealing with MongoDB, we use Monger, a library that just keeps getting better and better:

http://clojuremongodb.info/

In terms of my personal development, I have struggled to find a workflow that allows me to be efficient at the REPL. I am just now starting to use Daniel Szmulewicz "system" which I am hopeful about:

https://github.com/danielsz/system

Great article, thanks.

[from TFA]

    > This however does come with a cost:
    > - We cannot use Clojure’s concurrency primitives (futures/promises/agents).

Running my own large-ish Clojure service (CircleCI), I'm pretty convinced that futures are an anti-pattern in many situations. If your future is for side-effects, rather than for computation, the future call should probably be going on a durable queue being processed by a cluster of machines.

What are you guys using for durable queues? I'm currently evaluating Redis and RabbitMQ.

I wouldn't use either of those if you need durability.

What would you use? And why not RabbitMQ durable queues?

Probably referring to:

http://aphyr.com/posts/315-call-me-maybe-rabbitmq

But all systems like this have weak points, just look at the other systems aphyr has analyzed.

Kafka may be worth a look. It's the best message delivery system I've seen in the wild.

Anyone using Kafka with AWS? I'm concerned that it won't be partition tolerant enough to use it there.

http://aphyr.com/posts/293-call-me-maybe-kafka

We chose RabbitMQ, but we're achieving durability by storing messages in a backup database (CouchBase) that appears to handle partitions better. Still not ideal, and more complicated than we'd like.

This isn't helpful. It's a perfectly valid question that even provides insight into some possible candidates. The candidates might not be the best options, but they commonly serve this purpose.

What is it about RabbitMQ or Redis durability that you find lacking? What alternative do you recommend?

I've looked and have not found any alternatives that are better.

Did you rule out Kafka? If so, on what basis?

Pairs of RabbitMQ brokers, used in a Publish-One-Subscribe-Many fashion for availability.

Scaling RabbitMQ at SoundCloud: http://vimeo.com/69960105

Edit: usually with durable queues on SSDs

Edit: text version of the talk: http://blog.pivotal.io/pivotal/case-studies-2/scaling-with-r...

and here I am, staring down the barrel of Erlang. again

What type of issues are you having with Erlang?

That's a very interesting way of looking at it.

I don't quite get what you mean. The author's example is a GET request on some web service, are you arguing we should always be using queues instead of a request/response models when talking to external services?

No. My stereotypical use case is say, responding to an HTTP request that involves side-effects:

    (post "/user/foo" [request]
      (update-db)
      (future (send-user-email))
      {:status 200
       :body (render-page)})

What happens if email delivery fails? What happens if this box powers off? etc. Queues don't solve all your failure scenarios here, but they're a great first step.

I'm not sure that I understand. The future implementations that I'm aware of allow you to take different actions if a future succeeds or fails; in the above, couldn't you render a 500 if the email didn't send or the db update errored out? Furthermore, futures and work queues are not mutually exclusive. A future transformation could easily be the result of submitting work to a queue asynchronously; since you're talking about a separate service (possibly non-local), this submission itself isn't guaranteed to succeed.

I think arohner's point is that as soon as you use a separate work queue for asynchronous actions that are only executed for their side effect, then you don't need explicit futures anymore. Futures are building material for libraries and not something to be used directly in your code, because they need adornments that need abstraction.

As soon as you want to make the use of futures robust, you will basically start reimplementing parts of a work queue library.

E.g. futures assume they will be handled within the lifetime of the current process: they do not survive catastrophic process failure. If you want to overcome that assumption with persistence, you can either reinvent a wheel, or use a work queuing library that has already implemented this for you.

E.g. you can reinvent your own generic on-fail-retry or you can submit the work to a queue managed by library code that has already implemented retrying for you.

How are people getting along with core.async? I'm a bit surprised it wasn't mentioned in the article.

1) Give a kiss to your hand (left hand) 2) Say your crushes name. 3) Close your hand. 4) Say the name of a weekday. 5) Say your name.

6) Open your hand. 7) Paste this to 15 comments n the day you said In step 4 he/she will tell you they like you. If you don't do this you will have bad luck in the next hour.

Reply on news.ycombinator.com