Clojure is One Answer | Hacker News Reader

Clojure is One Answer

onbeyondlambda.blogspot.com

77 pointspron16 comments

As Herb Sutter recently pointed out, STM in Clojure is one answer to the problem of programming current hardware, but future architectures may not even have a coherent global memory model. I dare say Clojure will keep up though, as long as the JVM does.

Well, Clojure is one answer. Erlang's the other :)

But actually, I heard Rich Hickey has (or had?) some ideas for incorporating Erlang-style message passing into Clojure (at least for distributed, multi-node programs). And Clojure is well designed for that because its persistent data-structures are both immutable (and thus can all serve as safe messages), and are trie-based and so require only little copying upon modification and consequently relatively high-performance.

I'm not totally clear on the clustering situation, but I think this might be part of that:

https://github.com/liebke/avout

http://antoniogarrote.wordpress.com/2010/09/03/jobim-an-acto... (his more recent code for Gearman, Kilim)

Well, as other folks implementing this: haskell, F#, akka, scala actors have shown, getting a working implementation actors is not difficult but making sure green threads spin up quickly and all memory comes back, message passing large shared data structures, making sure the inbox doesn't explode, timestamping messages from multiple senders to one receiver, race/deadlock/starving, that devil is in those details, and where do you see the details managed relatively completely?.... OTP behaviors and the other erlang stuff.

I think here, as with many things i would characgterize as large architectural language features, it's important to see the limitations of STm (I think Duffy talked about htis in the STM/C# postmortem), the tradeoff between number of variables in the transaction, size/complexity of read/writes (e.g. putting collections or btrees in transactions), and number of threads reading /writing the TVar every second. GHC i would point out has some impressive STM benchmarks and I expect akka could do same but tweak a few things in your app and it will probably perform differently, very probably worse, depending on how much time you have on your hands.

I was thinkg in the context of the leaked Yammer/scala memo for an uncoming meetup, what if there were a dozen Rich Hickeys, or 2 dozen Simons for GHC, or whatever, their languages would still not be perfect, but the imperfections and myriad interactions of all these lang/runtime features would be clearly understood.

the issue of timestamping messages, guaranteeing send order, is one of the unresolved issues with BEAM and akka, and as far as I know, GHC and F#. You can guarantee send order for one receiver and one sender only. It seems to me that if you assume only local machines in physical proximity on a LAN, and assume away latency/partition, you could make some assumptions about message sending order from several sending processes (erlangese) to one receiver, or vice versa

Is Jobim still actively developed?

And while we're on the subject, whatever happened to Erjang?

>but future architectures may not even have a coherent global memory model

Just like a network cluster ? Clojure builds on persistent data structures and transactions, how those transactions are implemented can be abstracted, eg. www.avout.io

"Avout brings Clojure's in-memory model of state to distributed application development by providing a distributed implementation of Clojure's Multiversion Concurrency Control (MVCC) STM along with distributable, durable, and extendable versions of Clojure's Atom and Ref concurrency primitives."

Exactly. The important part isn't that Clojure has implementations of concurrency primitives, but that it has language semantics for defining when/how state may change.

The problem is, those primitives don't work in all cases. Believing otherwise would violate CAP. Avout is extremely impressive and I imagine I'll use it one day, but sooner or later you're probably going to want something like Rust's semantics _in addition_ to STM.

Clojure is brilliant because it even thinks about these problems, but even in the global mutable state arena where it excels there's things like concurrent hash tables where the advice is to use Java's existing solution.

I don't think CAP applies to the proposed scenario (architecture with large number of cores on single chip) unless cores can lose connection and each core needs to accept the update to guarantee consistency. I'm unclear on the whole "incoherent memory" architecture. I think that would be sort of useless, what I think will happen is there will be high latency path to do global atomic operations and low latency for cached/local, ideal for COW/STM architecture.

But it's difficult to know with fictional architectures :)

Well, strictly speaking the future's already here in the form of GPUs. It's just that we don't know what to do with them. But even if you ignore partition problems, synchronizing 1000 cores is a performance problem unlike synchronizing 8. It may be that we need explicit state that never leaves a processor.

But when do you actually need to synchronize all 1000 cores ? It's not like 1000 cores try to update same entity at one time, work will likely be map-reduce style (and there are plenty of things to do with it, speech recognition, voice synthesis, image analysis, etc.)

GPU have different issues, separate memory space, esoteric programming tools, doubles aren't IEEE compatible if available, no ECC. IIRC they aren't even suitable for things like ray-tracing, CPU implementations outperformed GPU based ray-tracers last time I checked. GPU's grew out of special purpose hardware and it shows, I don't think they are necessarily representative of the future architectures.

If you had a x86 or ARM chip with 100 cores and different instructions for atomic operations it would be a different story.

I agree, the most useful parallel implementation is nearly always mapreduce or pipes and filters. At the moment, Clojure's not that great for pipes and filters. The STM solution is brilliant, but atm it's all or nothing. The Rust model of being able to transmit something which ceases to be valid on the sending thread would be a useful addition to the current semantics. I think they're still thinking about the correct abstraction for event streams.

Azul nicly shows how you can have about 1000 cores with global memory. I for one would much rather build work with architectures like this, and so will most people and thats why I think its a model here to stay.

Here's the link to the original:

http://channel9.msdn.com/shows/Going+Deep/Expert-to-Expert-R...

If you watch the introduction, notice that when Rich Hickey motions the lisp syntax with his hands, he explains prefix notation in the viewer's perspective!

He's a fantastic presenter. I've found every single one of his talks extremely thought-provoking.

Reply on news.ycombinator.com