Software Engineering at Prismatic | Hacker News Reader

Software Engineering at Prismatic

blog.getprismatic.com

87 pointsfogus18 comments

"While we make heavy use of the core of Clojure, we don't use its concurrency primitives (atoms, refs, STM, etc.) because a function like pmap doesn't have enough fine grained control for our needs. We opt instead to build our own concurrency abstractions in Clojure on top of the outstanding java.util.concurrent package."

I wrote a web crawler in clojure at at&t and our team came to this same conclusion. First we tried to implement a custom atom, but finally using java.util.concurrent directly.

I'm not sure what this says about the clojure concurrency features, but I've found myself using them sparingly even when doing a lot of concurrency.

Rich Hickey is borderline genius in my book, and I really enjoy his philosophical approach to CS. Clojure is a fine language, but I frankly never bought the "concurrency issues" as a reasonable motivation for Clojure (over Java).

Fact remains that STMs (as originally developed by the Scala team in (yes) Java), "outstanding" [OP] concurrency, and idempotent/functional programming remains /well within reach/ of the seasoned developer in Java.

Also, unless I misunderstand their architecture (and it is single node ..), then the fact remains that in a distributed system, the "semantics" of interaction are not bound by the programming language behind box x, and are entirely a matter of the (remote) API you expose. STMs are a concern of a shared memory space -- once you go distributed, it is really borderline snakeoil to talk about the benefits of "functional" languages in creating remote functions.

Our system was both distributed and highly concurrent in the same memory space. We started off using atoms, but we needed visibility and control over the queue feeding the thread for the atom which lead us to using java.util.concurrent directly.

Our use case was fairly odd though.

For people outside the community this may seem like a surprise. But Rich Hickey has been saying for a long time now that Clojure explicitly does not try to emulate many of the tools found in java.util.concurrent for the very reason that they are quite good.

Right, and what makes me like clojure is the default of immutability. Even if I don't use their concurrency primitives heavily, the default to immutability makes writing concurrent code less scary.

"We tend to build our own libraries simply because much of the functionality of open-source code is locked up into monolithic frameworks and not easily re-usable."

Wow, I couldn't agree more with this concept of library vs. framework. While frameworks provide all kinds of up-front time savings, I've never used one that didn't end up getting in my way and require all kinds of deeply placed hacks to work how I (or my company/client) wanted.

I sometimes feel out of the loop because I'm not using all of the popular frameworks everyone else is talking about at the moment. So it's refreshing to hear I'm not alone in the opinion that time is saved in the long run by writing one's own small, modular, easily understood and reused libraries if none are already available.

I do feel as though small, self-contained code bases are becoming easier to find with the rise in popularity of sites such as GitHub and Bitbucket. So perhaps the trend towards less monolithic software can become more mainstream.

I had the same reaction. Fully agree with both you and the OP.

I'm most interested to know about the Flop library. This has always seemed a weak point of the JVM ecosystem. While there are many numeric libraries none seems dominant and speed suffers next to BLAS. Of course programming and using such a library in Java is also painful. This is a domain where functions really are by definition the right abstraction!

The CEO, bradfordcross, is amazing at distilling good engineering to principles. Software engineering for hackers.

Very much looking forward to the rest of the posts, and the release of these projects :)

where's the rss feed to subscribe to?

I'll put that up right now. Sorry it's a very new blog. Here's a direct link http://blog.getprismatic.com/blog/rss.xml

Any chance of getting back contents of Bradford's old blog http://measuringmeasures.com/ ? There were a few brilliant articles on becoming data scientist, learning statistics etc. It would be much appreciated.

that was a good one:

http://web.archive.org/web/20101102120728/http://measuringme...

Out of curiosity, is clojure used to do the front facing web components of prismatic as well or only the data/computation work?

this is an excellent question. i personally know two clojure teams, both use rails for their front-facing website - i believe they feel clojure's offerings aren't mature enough yet. sorry, i can't elaborate because I don't have any personal experience.

nerd crushing so hard this team and their work.

Great post. Very thought-provoking.

Reply on news.ycombinator.com