Comment by lazyjones | Hacker News Reader

Comment by lazyjones | Hacker News Reader

I really don't want to write this low-level code, or decide whether parallel or sequential implementations work best. Compilers targeted at particular environments can do this better.

My main takeaway from this is that most programming languages still fail at allowing the programmer to specify intent where a particular implementation is not desired - I cannot use the mathematical sum notation for the accumulator example and let the compiler choose the best implementation, instead I need to implement a particular array traversal or partitioning. And in situations where high level language constructs would allow a certain flexibility w.r.t. implementations and parallelism, usually this isn't taken advantage of by the compiler/interpreter and programmers even rely on that fact (e.g. map in Perl could work on list elements in any order or in parallel, but all hell would break loose if it actually did at some point because people are now used to sequential processing).

Actually, I think what you're really asking for is the ability to write sequential code and have the compiler work out if it could be parallel. It's true that the reducers implementation runs in parallel, but the API is actually agnostic.

Reducers also provide higher level constructs like filter, map, cat (concatenate), take-while, and of course reduce.

> what you're really asking for is the ability to write sequential code and have the compiler work out if it could be parallel

Not really. I don't care how I have to specify the desired computation, as long as it isn't excessively verbose and doesn't require me to nail down implementation details I don't care about - for example, a for-loop (from 1 to 1 million as in the accumulator example) specifies a particular traversal (index 1 to 1 million in ascending order), but for an implementation of "sum" I only need "for each element" (no particular order) semantics. I don't want to even think about whether by using such an (overly specific) loop, I'm making the code difficult or impossible to parallelize (in the presence of some side effects, for example in C, it will likely be).

Actually, no, for sum you need three things: how to start (0), how to accumulate (plus) and how to combine two reductions (plus again). This is exactly what fold doe

Equally, for filter, it's empty list, conj or not and concat. (Just to demonstrate that sometimes the combine is different from the accumulate stage.)

> Actually, I think what you're really asking for is the ability to write sequential code and have the compiler work out if it could be parallel.

Nah; the ability to write parallel code and have the compiler work out how to make it sequential-when-necessary would be better. Like JITing an agent-based system into bits of single-threaded procedural code.

Agree. I also prefer high level operations that are ought to be optimized by the runtime. Python list comprehensions let you do things that are closer to the "mathematical sum notation" without you having to specify the implementation details. I am not sure how optimized it really is though :)

It's not really optimized. The list comprehension's body gets executed in series, and even if it were threaded, the GIL would forcibly serialize the thread execution while breaking all the code out there that expects in-order processing.

The reducers library is actually rather declarative. Operations that are unordered are executed in parallel, whereas those that need to be ordered are executed in a single thread.

Reply on news.ycombinator.com