Comment by arximboldi | Hacker News Reader

Comment by arximboldi | Hacker News Reader

Parent Full thread

There are many things to love in Clojure, like core.async, the lispyness, etc. But for me, their biggest breakthrough is with no doubt the data-structures.

Clojure default of immutable hash maps and vectors was really bold. They had to innovate by giving a twist to Bagwell's great research on data-structures, and they managed to build what in my opinion are the first set of general data-structures suitable of representing traditional hierarchical object-tree data-models immutably, at the scale of large interactive software.

Without these, the single-atom architecture that now is popular as ever with Redux, Elm, etc. would not be possible. The flourishing of functional programming for interactive software that followed shows to me that the biggest impediment before was not the lack of functional ways of expressing interactions (the Haskell people have devoted a lot of notable effort in the FRP camp). Instead, just having efficient data-structures to write persistent document models, escaping the rigidity of lists and rb-trees, allows you to write big interactive sofware in a functional manner even with the most dumb abstractions (which is a good thing, KISS!) and even stupid language (JavaScript!).

This has inspired me to work on a generic implementation in C++, which also has experimental Python and Guile bindings: https://sinusoid.es/immer/ https://github.com/arximboldi/immer/

In the past I also did some other work on bringing Clojure Transducers to C++ (https://www.youtube.com/watch?v=vohGJjGxtJQ&t=1614s) but I know consider it a much less important effort---without the data-structures the other parts of Clojure feel almost like just sugar.

EDIT1 -- Sorry for the shameless plug ;)

EDIT2 -- Grammar

> In the past I also did some other work on bringing Clojure Transducers to C++ ... consider it a much less important effort---without the data-structures the other parts of Clojure feel almost like just sugar.

Very few clojure programmers consider Transducers very important even in Clojure. While they're convenient for implementing Clojure's stdlib and they have some nice properties, they're very difficult to work with. It takes a very experienced clojure programmer to use transducers. Heck, I know some famous clojure developers who admit they themselves don't feel confident using them.

But the main challenge for them is that transducers just scream for some basic type validation so that you don't accidentally chain together nonsense. C++ could provide that, and you'd end up with all the same properties. In particular, they offer something like half the monad laws and a more easily composed form of CPS (how your continuation is called is more predictable).

I'd think that might be a really valuable thing in C++, as it's a construction that reproduces stream fusion.

Know what you mean RE transducers. I still haven't come to a point where using the transducer pattern is more convenient than just (partial map f) tbh.

Yet to dare clojure spec but it feels likely that the next mind blowing journey will be therein.

Writing your own transducers is hard and error prone, but I think using and composing the ones given you by clojure.core is actually rather easy (Especially using the 3 arity of `into`).

I don't use so many `->>` anymore thanks to them.

I dunno. Except for one very specific project, I've yet to see a "Wow!" use case for transducers. The only place where they really shine is working with channels, but I think channels are a big design mistake (after extensive work trying to tame them).

Interesting! Why do you think they are a big design mistake? I can understand how they can be misused, but I find myself missing them in every language I used that doesn't have them...

So either I'd rather talk about the COMPUTATION and resource safety rather than the channel, as in Pipes/Conduit/Streaming from haskell.

OR, I'd rather deal with a direct actor model a la Erlang, Pony and Cloud Haskell.

Go's experience is so goddamn miserable it should just put the entire notion of async channels as a higher level programming library to bed. As a low-level primitive, it's okay but not fundamental.

You might find this work on composable well-typed transducers in Idris interesting: https://deque.blog/2017/08/04/transducers-in-idris-2/

Yeah, I agree!

Actually one of the things I found during my work on transducers is that actually they seem to me more implementable in C++ than in Haskell. For example, you can implement state-less transducers with almost zero overhead, something that in Clojure is definitely not true (so the standard transducers do hide state inside the closure of the reducing functions). It is also easier to do n-ary transducers for zipping and unzipping and everything, as you intuit, is type checked.

Still, I agree that transducers are most useful when you have libraries that make sense of them. For example, you could implement a lot of "Rx" using transducers. Ideally I see a user never writing a transducer, and occasionally passing one to some library that provides "higher order" reactive collections (channels, observables, etc.) that can be transformed with transducers. For example, my motivation to implement them in C++ was to build "observable/reactive lenses" (something like "cursors" in Om) [1] where you could use transducers to specify how you "zoom" into the virtual sequence of state values...

Anyways, they are an interesting coding tool, but they don't really influence the architecture of the application.

[1] atria::funken https://github.com/Ableton/atria

> they seem to me more implementable in C++ than in Haskell

If you don't mind the overhead, it's ContT wrapping some base monad. Or you could make it integral. That is more specific than transducers, I grant. But there's not much they can do that ContT over like Array does not.

I think it's too bad they were invented in Clojure, because they're brutally hard to use right and write in clojure.

I have some interesting work I could share in private on using them to provide stream fusion properties to pipelines of channels. Hit me up on keybase if you'd like to see. I don't want to open source it because then people will expect me to explain them and maintain them and I don't feel like it :)

Sorry, I actually meant "than in Clojure". You are right about the ContT. I'm actually curious about your transducer based stream fusion stuff... I'll hit you up later about it :-)

Do you know of a good resource for reading about their data-structures, especially anything Haskell doesn't have?

Any book on clojure and website too will go over them. They are fundamental to using clojure.

I don't know enough about haskell to say what it does or doesn't have in relation to clojure. However, it's not simply the data structures, but how the language is built to interact with them from a programmers perspective.

From a user interface to the language point of view, I find the ways of interacting with maps to be very nice.

In particular the dual purpose of "keywords" to serve both as a good index/key in said hash maps, but also as callable look up function for the same key in a hash map leads to simplified code.

This, coupled with the extremely high usage of those maps in typical clojure code, makes a big difference.

To expand on that a little, "keywords" are symbols that begin with a colon, which I believe is the same as common lisp. :for :example :this :is :a :sentence :of :keywords. They are commonly used as the key in a map. Which has a literal syntax like (def map1 {:a 1 :b 2 :c "taco"}). To pull a value out, you can evaluate (:c map1) which will return "taco".

They are hash maps based on work by Philip Bagwell, and I think a lot of languages have versions of this now. At least scheme and Common Lisp do. But I don't believe they have the nice literal syntax and interaction syntax.

Many Lisp traditionalists are bothered by the use of curly braces for maps and square brackets for vectors, however I'm a fan. Without this bit of syntax, you have to add verbosity to the language to differentiate defining a map from doing a lookup, etc.

For resources, Clojure for the Brave and True is pretty popular. Also the online community is quite friendly, although I have noticed some trends toward "the one true way" of doing things that puts me off a little.

>Many Lisp traditionalists are bothered by the use of curly braces for maps and square brackets for vectors

I have definitely seen this. I'm not sure that I understand this sentiment. The only thing that Clojure is missing is the "system" feel that Common Lisp provides, with things like character macros, symbol macros, etc.

That being said, I definitely appreciate how thought through Clojure is, especially when compared to CL. Several things are missing/don't feel well thought out in CL (notion of a `calleable` type is gone, `nth` isn't defined on vectors or strings (but you can still `length` them!) nor is `first` and `rest`) That provides a great deal of friction.

> notion of a `calleable` type is gone

You can do this using the metaobject protocol and defining a class with a metaclass of FUNCALLABLE-STANDARD-CLASS, for example, given:

    (defclass c ()
      ((x :initarg :x :accessor x))
      (:metaclass funcallable-standard-class))

    (defmethod initialize-instance :after ((c c) &key)
      (with-slots (x) c
        (set-funcallable-instance-function
         c
         #'(lambda ()
             (format t "~&I'm #~a" x)))))

Then (funcall (make-instance 'c :x 3)) will print "I'm #3".

> `nth` isn't defined on vectors or strings (but you can still `length` them!)

ELT in Common Lisp is generic over sequence types. NTH is specific to lists. If you want to write a function that is generic over sequences, use ELT. If you know that you have a list or an array, and you're writing something performance sensitive, you can avoid type dispatch overhead by using NTH or AREF instead of ELT.

> nor is `first` and `rest`

You can avoid that friction a lot of the time by writing in terms of MAP, REDUCE, REMOVE-IF-NOT, and so on, as well as LENGTH and ELT, which are all generic over sequences, or doing (iter (for x in-sequence sequence) ...) or whatever.

But if you really want to use FIRST and REST to walk through a vector, you can do (coerce vector 'list) first to convert it. Coercing a list to list will just return that list, so that version would work for either, too. I admit, there's no real reason I know of (aside from lack of demand) that there isn't a predefined function like:

    (defun vector-rest (vector)
      (make-array (1- (length vector))
                  :displaced-to vector
                  :displaced-index-offset 1))

Then if it bothered you you could do something like

    (defun head (sequence)
      (elt sequence 0))

    (defun tail (sequence)
      (etypecase sequence
        (list   (rest        sequence))
        (vector (vector-rest sequence))))

Common Lisp had a lot thought put into it, but the design goals were different. One of the design goals was to support efficient compilation given various constraining dimensions: the need for speedy compilers vs. the need for delivery of applications on standard hardware. Thus a Lisp compiler should be able to generate fast code, even by using whole-program compilation. Thus the language has to take care of a wide range of possible implementations, not just one on a particular virtual machine.

Clojure OTOH has a relatively straight-forward compiler written in Java, only targetting the JVM in a standard way. The Java/JVM infrastructure for code generation (JIT), low-level optimizations (JIT), memory management, application building and delivery could be reused. With all its features and limitations (for example no TCO, fixed JVM instruction set, ...).

Common Lisp also not only provides some generic operators, but also lots of non-generic operators. One reason for that is the will for backwards compatibility (which was another goal in the Common Lisp design) and the idea that something like Common Lisp is also a low-level languages or provides at least a layer of low-level stuff, functions you would use to implement higher-level stuff.

Common Lisp is the next step in the evolution of Lisp I to Lisp 1.5 to Maclisp to Zetalisp. Thus it contains much of the core Lisp functions, stuff you can read about it the Lisp manuals from 1958 and 1960. People did not want to throw their code away, they wanted a modernized Maclisp. Thus one could port a 100kloc Lisp program from Maclisp or Zetalisp to Common Lisp without too much effort. The main operations were there.

Clojure OTOH has almost no backwards compatibility to earlier Lisp dialects. Programs can not be ported from Lisp to Clojure in a useful way. They have to be rewritten. But that was okay, since the main target were not Lisp developers, but people who wanted a language with Lisp influence in the Java/JVM eco-system (or possibly other eco-systems too).

Thus Common Lisp looks to you like it has less thought put in, where people actually struggled to be backwards compatible and modernize the language at the same time.

So Common Lisp has lots of non-generic functions, a layer of generic non-extensible functions (like the sequence layer) and a mechanism for extensible generic functions (CLOS). Something like CLOS is implemented on-top of the non-generic Lisp, with a few parts using implementation specific mechanims. One does not need to use the primitives of a different host language, because Common Lisp is already capable of hosting with its primitives.

> Many Lisp traditionalists are bothered by the use of curly braces for maps and square brackets for vectors, however I'm a fan. Without this bit of syntax, you have to add verbosity to the language to differentiate defining a map from doing a lookup, etc.

I agree with you, in the sense that they are useful, however the complain of "Lisp traditionalists" will be that in any case, if you need them, you can add them to Common Lisp by using reader macros. So in CL the curly braces and brackets are "free" to be used (i.e. define what they should do) in any way you want; on Clojure they are already taken, so it feels a bit uncomfortable.

> To pull a value out, you can evaluate (:c map1) which will return "taco".

Not (map1 :c)? A bit odd.

Both work I believe, but the idiomatic way is keyword first.

Here is a good resource on Clojure's persistent vector type: http://hypirion.com/musings/understanding-persistent-vector-...

Hash Array Mapped Tries are what underly the default maps in Clojure: http://lampwww.epfl.ch/papers/idealhashtrees.pdf

They are pretty amazing. Of course there are implementations in Haskell now, but the reason they are especially cool in Clojure is how well they integrate with the language and its other features.

I think Bitmapped Vector Trie [1] might be some novelty introduced by Clojure.

[1] https://stackoverflow.com/questions/8844707/how-is-a-bitmapp...

Folks have linked you to the paper and mentioned tutorials.

You might also Okasaki's Purely Functional Data Structures, which ends up in the citations for a lot of these.

Okasaki's work is really interesting, and it is quite mind-blowing the theoretical properties he achieves by exploiting laziness and so on. At the same time: those data-structures have one fundamental flaws they have high constant factors, specially outside of Haskell. That was Phil Bagwell + Rich Hickey revolutionary contribution in my opinion... finding data-structures that are rather simple yet but work well in practice because they make the right engineering trade-offs and work well with CPU caches, etc.

Here's a blog post[0] from the guy who implemented Lean Hash-Array Mapped Trie (HAMT) maps for clojurescript. It's really cool stuff!

[0] https://bendyworks.com/blog/leveling-clojures-hash-maps

Search "Phil Bagwell", most of his papers are quite interesting.

If you allow me another shameless plug, I also compiled a good bunch of references in the "Related Work" section of my own paper on immutable vectors: https://public.sinusoid.es/misc/immer/immer-icfp17.pdf

Reply on news.ycombinator.com