Building Clojure Services at Scale | Hacker News Reader

Building Clojure Services at Scale

blog.josephwilk.net

149 pointsr4vik21 comments

One thing I've noticed with my own Clojure service at scale, is that eventually, I want to significantly reduce my use of futures entirely.

When you're getting started with Clojure, it's easy to do everything that needs to happen "later" or "in the background" as a future. Once you go to production, you realize that about half of those futures should have been jobs going into a queue somewhere, with logging and guaranteed delivery, retrying on another machine, if necessary.

I haven't gotten to the point of writing that library, but I probably will, soon.

Hey Allen, at Factual we're working on something in that vein: https://github.com/factual/skuld. As you might expect, doing it right takes something a bit more ambitious than just a library.

If you're interested in collaborating, or just have thoughts on how it could be more useful for you, let me know.

I've had great success using Lamina for in-process message queues in lieu of futures, although this does nothing towards making your system distributed.

In process queues are a good abstraction, but that doesn't really protect you in case of your app crashing, which I think is the concern more than being distributed. I may be misunderstanding though.

Edit: I was already a big fan of SoundCloud, very pleased to find out they use Clojure.

I think the point was that it's hard to make something failure-tolerant without being distributed. A locally persisted queue would avoid certain failure modes, though.

Yes, I shed a tear as I had to remove all the futures. Disappointment I could not get multiple thread pools and scheduling with Clojure's concurrency primitives.

With Ghostery on, there's no content on this page unless you unblock the SoundCloud widget.

In this case, it's definitely worth unblocking. :-)

Thanks. I'll check it out. Was planning on skipping it because of Ghost.

Thanks for the note, I've ensured Ghostery only blocks the SoundCloud widget and not the entire post now.

This is the first I've heard of the "Scala runtime". Isn't it compiled to bytecode and usable as a jar dropped in anywhere on the JVM possibly with some name mangling if you don't design your public interface to be pretty for java based projects?

That's right, its really just a library. Also, Finagle already has an idiomatic Java API so using it from Clojure would not require use of any mangled names.

Thanks for those replies. I've updated that section to better reflect Scala just being a library dependency.

I'm looking forward to playing with Finagles Java API as at some point I'm going to need to connect our Finagle projects with our Clojure services.

> MainGenericRunner will look at the file and determine if it's a script (.scala) or class file (.class). A class will be passed straight through to the JVM (+/- some classloader trickery) while the script is passed into the interpreter first. It is I think just a dispatcher.

im pretty sure he means the standard library, e.g. data structures. Sharing Scala and Clojure code means converting your data structures at the boundaries.

This is true for Java and Clojure as well. In Clojure you have to choose between using java data structures verbatim or else converting them to clojure ones.

See the scala docs for JavaConversions to see how to make scala collections look like java ones.

This is becoming more and more common for Scala projects that want to build libs in Scala and only export a Java interop API (as opposed to pure Java at the bottom and building Scala on top). E.g. https://github.com/apache/kafka/tree/0.8/core/src/main/scala...

The real difficulty comes when you want to use multiple scala libraries in a project which are compiled against different scala runtime versions.

I live in scala land, but I prefer java libraries for this reason.

If you want writeups on scala/java interop, the Manning "in Action" and "in Depth" books have excellent summaries. From scala you need to worry about java annotations, static members and "throws" exception declarations. Java private /protected etc are checked at runtime and compile time. (That's just me copying items out of the "In Action" ToC, but everything's very well documented).

For clojure/java interop, there's also good writeups in 2 Manning books, and also the OReilly and Pragmatic books. It looks like combining them (Scala/clojure interop) would take some plumbing, depending on the specific libs involved but i havent' tried. (And the Manning authors are all pretty far along on 2nd editions of both "Joy" and "In Action")

http://www.manning.com/fogus2/

I've used Hystrix to great effect before, but am now really curious about mixing it with LMX Disruptor aka Reactor and it's clj bindings ( https://github.com/clojurewerkz/meltdown ). Anyone have any experience doing so?

That looks very interesting, will have to play!

I was recently looking for clojure bindings. Meltdown is the only one I could find that is currently maintained but it doesn't appear to expose consumer barriers anywhere. Without them the disruptor is just a lockless ring-buffer. The barriers are what allow building multi-threaded pipelines on top of it.

Reply on news.ycombinator.com