Hacker News Reader

Comment by dustingetz | Hacker News Reader

Not sure I understand - so Clojure is getting first class support for lazy collections and curried combinators? Or am I missing the important part?

View on HN

Many Clojure abstractions need things like map, filter, concat, etc. Reducers are eager, lazy-seqs are lazy, and core.async channels are lazy and push-based. This is a way to unify all these abstractions so that you can write a single map/filter/concat transform once, and use it in many different ways.

Isn't there some overlap between this and extend-protocol? To my vague understanding this is from the functional composition angle instead of a type, but otherwise don't they accomplish the same thing?

> This is a way to unify all these abstractions so that you can write a single map/filter/concat transform once, and use it in many different ways.

I don't understand why these things aren't all unified in the first place, it's just function composition we're talking about here right?

Edit: I mean it's clearly not just simple function composition, but I don't understand why not

    (def xform (comp (map inc) (filter even?))) ; xform is a transducer
    (defn xform [aseq] (->> aseq (map inc) (filter even?))) ; xform is a fn

Why do these have to be different things? Why is there a need for machinery to do what fn composition is supposed to do?

Your typical definition of map, filter etc includes concrete usage of e.g. lists. These don't. Transducers are not just currying or partial application of the map function over lists, they isolate the logic of map, filter etc from lists or any particular context, allowing them to be used in very different (e.g. non-collection) contexts.

By "map in a non-collection context", what would be an example? Say, mapping a boolean negation function over the bits of an integer, to produce its complement? How does that look with transducers?

Well, the most obvious value to me is that you can create algorithms that defer their action- So instead of having a function that operates on a collection, you can have a function that will operate on a collection at a future date, even if the collection does not yet exist. (that's what the clojure reducer library does)

(And if you say "deferring an action is just another term for 'functions'" then that's exactly the point, lifting more algorithmic logic into composable functions)

Alan Perlis Epigram #2.

> isolate the logic of map, filter etc from lists or any particular context

so Functors (fmap)? fmap still uses regular old fns, no machinery.

http://clojuredocs.org/clojure_contrib/clojure.contrib.gener... http://learnyouahaskell.com/making-our-own-types-and-typecla...

I feel that same confusion as you. In Haskell terms, what is it Rich have implemented or perhaps even invented?

I'm not sure I grok this, but I think that the main points are:

- reducers can work on tree structures, and thus can exploit parallelism. This would be like using a map that requires only the Foldable typeclass

- In Haskell you have stream/vector fusion, but it's not obvious to know when ghc will actually exploit it, you might want to use something like Control.Monad.Stream or Data.Vector. In theory it might be generalized to all Foldables, but in practice for now it might be a good enough compromise to stick to a limited number of types (the one that support such transducers)

So: nothing terribly new/groundbreaking, but it might bring something like stream fusion to the masses (of clojure developers :P )

Comparing transducers to stream fusion is far narrower than the scope of their applicability.

They are more comparable to Oleg's Enumerators (http://okmij.org/ftp/Haskell/Iteratee/describe.pdf), in that you compose a series of computations and then push data through them. The type signature is similar:

    type Iteratee el m a -- a processor of 'els' in the monad 'm' returning a type 'a'

    type Enumerator el m a = Iteratee el m a -> m (Iteratee el m a)

The Enumerators library is complicated by the presence of monads and by trying to automatically close handles when the stream is processed. In some ways it seems that the goal of solving the lazy IO problem led to missing a much simpler abstraction. Transducers seem to be simpler and more focused on just abstracting stream/map-reduce computation.

While not mentioned in the blog post, the transducers implementation supports both early termination and result completion/flushing/cleanup.

there does seem to be some overlap with stream fusion -- which was all about exploiting the optimization opportunities when separating the collection operation from the kernel performed on each element.

We called the bits in the middle "step functions" , which could be combined with "consumers", "producers" and "transformers".

And the algebra generalizes hugely (not just collections) but to things like concurrent processes, data flow programs etc.

http://metagraph.org/papers/stream_fusion.pdf

Things to think about in a non-Haskell settings: how do you prevent reordering side-effects? Can execeptions/non-termination being reordered be observed?

> they isolate the logic of map, filter etc from lists or any particular context

Is it correct to say that the relationship between the new forms of map/filter/etc, reduction functions, and transducers is something like this:

"traditional" (map f l) is equivalent to (foldl (mapR f) l []), where (mapR f) is the "reduction function" corresponding to map. (mapR would still be list specific)

the new (map f) is a transducer that takes a reduction function and returns another reduction function; given idR, the "identity reduction function" for foldl such that (foldl idR l []) = l, ((map f) idR) = mapR.

Furthermore, given reduction functions mapRR such that (foldr (mapRR f) l []) == (map f l) and idRR such that (foldr idRR l []) = l, then ((map f) idRR) = mapRR. (Because all the list-specific things in the output reduction function come from the input reduction function, the transducer (map f) doesn't need to know anything about lists, so can be used in other contexts as well -- one minor-but-perhaps-easier-to-illustrate aspect of that is, even when used with lists, (map f) is independent of the folding direction, unlike the reduction functions mapR and mapRR.)

(I know this is barely even scratching the surface of the applications, but I'm trying to confirm that I've got the concept right.)

Yeah, when looking at core.async transformers and the reducer library, it seemed clear that there was some generality in there that could be factored out. Glad someone smarter than me figured out what it was.

I think this is just announcing that the Clojure core libraries are gaining another arity for many functions specifically for this usage so that you don't have to use (partial) or the short-hand #() syntax to use this pattern. I assume the goal is more community awareness and cleaner syntax for the re-use of this pattern.

No, (map f) is not curried map. It returns an entirely different thing - a function of reducing function to reducing function, aka a reducing function transformer, aka a transducer.

Providing the signature of this new `map` function (e.g. as in Haskell's fmal http://www.haskell.org/hoogle/?hoogle=fmap), would certainly go a long way towards helping people understand what this `map` does.

These sigs are for the arities below only.

map f: (a->b)->(x->b->x)->(x->a->x)

filter pred: (a->bool)->(x->a->x)->(x->a->x)

flatmap f: (a->[b])->(x->b->x)->(x->a->x)

etc.

So, a transduceMap implementation would be this I guess?

  transduceMap :: (b -> a) -> (acc -> a -> acc) -> (acc -> b -> acc)
  transduceMap f = \reduceFn -> \acc el -> reduceFn acc (f el)

lambda added for clarity (no pun intended), however types are easier to match when using this syntax:

  transduceMap :: (b -> a) -> (acc -> a -> acc) -> acc -> b -> acc
  transduceMap f reduceFn acc el = reduceFn acc (f el)

OK, maybe we're getting somewhere. Let me try to write this `map`, just to see. I'm using Scala, so I can put some types, and have the compiler yell at me if I'm doing something overtly wrong (I need all the help I can get!).

For clarity, let's define a type alias for reducers:

    type Reducer[X, A] = (X, A) ⇒ X

Let's define `map` to match the type definition you provided. And with that type definition, I only see one way in which the function can be implemented. So it must be:

    def map[X, A, B](f: A ⇒ B): (Reducer[X, B] ⇒ Reducer[X, A]) =
        (redB: Reducer[X, B]) ⇒ (x: X, a: A) ⇒ redB(x, f(a))

How can I use this? Let's try the following:

    def addup(zero: Int, a: List[Int]) = a.foldLeft(zero)(_ + _)
    def parseList(a: List[String]) = a.map(_.toInt)

    map(parseList)(addup)(1, List("7", "8"))

This returns 16. OK, parsing the list, and adding up starting from 1. But it doesn't look to me like `map` implements anything like the usual semantic of map. It just converts the data structure, and applies the reducer. What am I missing here?

Reply on news.ycombinator.com