Clojure's Solutions to the Expression Problem | Hacker News Reader

Clojure's Solutions to the Expression Problem

infoq.com

158 pointsadityaathalye27 comments

I miss the days when productivity booster posts were all about REPL, functional programming, composability, immutability.

I just wish more people appreciated those things at all. What's the name of the rule where everyone with a sufficiently large codebase has a partially implemented, buggy time oriented data log somewhere and it's too bad people didn't realize that sooner. Datomic et all are so cool and solve so many interesting and pervasive problems but people would rather just mangle those bits themselves. The glory of refactoring a large, purely immutable piece of software is so glorious and easy. The wrapper/adapter hell that comes out of the Expression Problem is just one more thing that people just assume is unavoidable and the amount of gross code that falls out of that is so painful.

"Henderson's Tenth Law" :D

  *But beware. Once you see, you cannot un-see the fact that…*

  Any sufficiently complicated data system contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of a bitemporal database.

  — Henderson's Tenth Law. (= henderson https://github.com/jarohen)

I blogged about it here: https://www.evalapply.org/posts/poor-mans-time-oriented-data... (discussed here recently: https://news.ycombinator.com/item?id=44583790 ).

(edit: add links)

I enjoyed reading most of that.

One thing to watch out for with immutability is that if you're dealing with personal information about people, immutability is probably illegal. You must be able to forget information, and not just simulate you've forgotten it.

I don't how universal this is at the moment, but I think it's likely to be more universal in the future.

Thank you for the kind words, and feedback!

Yes, controlled excision is a (necessary) capability of all event-sourced systems and/or immutable object stores, i.e. it's quite universal already. All such systems have to build in some sort of special-case mechanism to excise facts, not merely redact (~tombstone) them.

But data-destruction superpower must be used sparingly, with care. See: git's documentation for --force-push, for example (and bemoan what popular git forges force us to do on a normal basis).

Back to DB-land...

Datomic does it this way: https://docs.datomic.com/operation/excision.html

Excision is the complete removal of a set of datoms matching a predicate. Excision should be a very infrequent operation and is designed to support the following two scenarios:

    Removing data for privacy reasons
    Removing data older than some domain-defined retention period

Excision should never be used to correct erroneous data and is unsuitable for that task as it does not restore any previous view of the facts. Consider using ordinary retraction to correct errors without removing history.

XTDB calls it ERASE: https://docs.xtdb.com/reference/main/sql/txs.html#_erase

Irrevocably erases documents from a table, for all valid-time, for all system-time.

While XTDB is immutable, in some cases it is legally necessary to irretrievably delete data (e.g. for a GDPR request). This operation removes data such that even queries as of a previous system-time no longer return the erased data.

> The wrapper/adapter hell that comes out of the Expression Problem

Yeah, I got lucky in my work (see sibling comment https://news.ycombinator.com/item?id=45207880).

For me, it was PageObjects all the way down, which "just composed".

Something like this:

  (defprotocol IPageObject
    "Each PageObject MUST implement the IPageObject protocol."
    (page-object [this])
    (exists? [this])
    (visible? [this]))

And then an implementation like this:

  (defrecord Checkbox [target-css]
    IPageObject
    (page-object [this]
      this)
    (exists? [this]
      ;; webdriver check if target-css exists
      )
    (visible? [this]
      ;; webdriver check if target-css is visible
      )

    Selectable
    (select [this]
      ;; webdriver select the target
      )

    (deselect [this]
      ;; webdriver undo selection
      )
  
    (selected? [this]
      ;; webdriver return true if target selected
      )  

    Value
    (get-value [this]
      ;; webdriver return current selection state
      (selected? this)))

Thanks to Records (stateless, type-identified, hash-map semantics), the PageObject model composes arbitrarily. i.e. A Page is itself a PageObject containing a tree of arbitrarily nested PageObjects... i.e. PageObjects all the way down.

Solid gold!

(edit: x-link to sibling comment)

This example just scratches the surface. I don't think it's showing any Clojure-magic per se, b/c in effect what you're showing is just "implementing an interface". You can do that in most languages.

The magic of protocol/records is that they work across library boundaries. A library may provide a record - and then you can extend the record with new protocols. Key is that it's all without needing to explicitly creating new agglomeration types.

You can take some Dog record from some pet-simulation library and then `extend-type` it with the IPageObject protocol and make the Dog record now something that can be displayed on a webpage

The magic of Clojure is that if a problem can be solved with an interface Clojure lets you solve it with an interface. It doesn't have any magic to show off, it is just a relentless implementation of a lot of basic good ideas.

???

I don't understand. Did you not read my comment?

I just explained how your seemlessly extend records across library boundaries. It's not just solving with interfaces

I don't agree with your characterisation of what the magic of protocols is. A protocol is the lightest possible thing that captures how two things can interface. That is magic in a way, but it is also the end of it, Clojure does a masterful job of keeping concepts tidy. In terms of what you can do with protocols, you can indeed do that in many languages. Clojure is pretty smooth but there is nothing stopping anyone implementing any interface on any object in most languages; build some sort of adapting whatever. It is rare enough in practice that the extra dev work isn't a major problem.

And as a bonus observation, any time your example of how to use a programming feature involves animals it is a fake use case. It is a bad sign because it suggests that the benefit doesn't have a real world use.

> Clojure is pretty smooth but there is nothing stopping anyone implementing any interface on any object in most languages

I mean, this is false for Java. Creating a wrapper can end up being a big mess if you need to expose the functionality of the inner type. And if you need to preserve the actual underlying type, you end up needing wrap/unwrap all over the place. I think you're underestimating the cost of needing wrappers.

I would also argue that "it's rare enough in practice" because people just avoid doing it, either just avoiding using a library because it would be too much work to wrap, or more likely, just writing directly to the implementation instead of using an interface.

Mmm, I don't think what you're saying is true?

I'll be honest I haven't touched Java in a long time, and I'm haven't had to extend my Clojure code this much before (but it's nice to know that I can - and use protocols with confidence)

Say LibraryA has some record..

LibraryB and LibraryC `extend-type` the record and implement additional functionality by implementing protocols..

Now you want to us both libraries in your application and use this record with both extensions. In Java this is forbidden due to diamond dependencies restrictions.

You can use multiple protocols in Java, but you can't have the protocols implemented (there are default methods but they don't have access to the Record's state)

Is there some way around this..? As far as I understand there isn't.

Furthermore, all the extensions need wrappers and need names. Nobody is doing this b/c you'd go nuts. But in Clojure it'd be a natural way to extend a library.

For instance

    (defprotocol MyProtocol
      (foo [x]))
    
    (extend-type clojure.lang.PersistentVector
      MyProtocol
      (foo [x] (str "Vector of size " (count x))))

    (extend-type clojure.lang.PersistentList
      MyProtocol
      (foo [x] (str "List of size " (count x))))

I can keep using vectors and lists as before and have new functionality. I don't need a new wrapper to think about

I can't claim to know much of anything about Java. But this sounds like it might be linked to Java's type system and Clojure is an untyped language. From what I know of typed languages I don't see how a Clojure style interface could be implemented in a typed language or why someone would want to do that if they have chosen a typed language. So I don't understand the point and I don't know Java.

I'm fairly confident that Java is technically capable of doing anything Clojure can since in the extreme case someone could implement a Clojure library callable from Java. In a similar way to how the Python community claims to do machine learning despite all critical code actually being in C++.

Okay... I thought you had something actually to contribute to the conversation

You start off incredibly dismissive

> Clojure is pretty smooth but there is nothing stopping anyone implementing any interface on any object in most languages; build some sort of adapting whatever

Then i explain how it seems not possible.. I mean maybe I'm wrong and missed some angle. I'm really willing to learn and change my mind here. And yes, the dynamism is part of the whole point.

And then you just handwave it away, making some vague analogy to Python and ML.. and don't actually have any technical input at all. What a frustrating waste of my time

> Okay... I thought you had something actually to contribute to the conversation

> You start off incredibly dismissive

You seem to think it is polite and reasonable to open a comment by being dismissive. I don't even think I was particularly dismissive, that would be reading a lot into two mildly disagreeable sentences.

> I mean maybe I'm wrong and missed some angle.

Well the conversation has manoeuvred to somewhere we neither of us are current on the topic, which is Java. I'm not sure why we're talking about Java, you can call Clojure libraries from Java, so there isn't anything Clojure can do that Java can't. Clojure is implemented in Java and Clojure protocols can be implemented on Java objects by proxying them.

It is difficult to say that Clojure has some magic trick Java doesn't under those conditions. Java has really good access to everything Clojure offers. Clojure's advantage is it organises what Java offers in a much more Senior-Engineer-friendly way that radically minimises complexity.

Sounds like type-classes from haskell/scala? Or is that a different thing?

I'm not familiar with Haskell so it's hard for me to say. A cursory look at the wiki.. I get the impression you'd need to explicitly create a new type class? In a dynamic language you can just "dynamically" extend the record (which is sort of like a "class) with new interfaces without making a new class type.

Hopefully that helps.. Sorry if it's not exactly helpful

The protocol example at the end shows how it's done

https://eli.thegreenplace.net/2016/the-expression-problem-an...

If you write something like

    (extend-type BinaryPlus
      Evaluatable
        (evaluate [this] (+ (evaluate (:lhs this)) (evaluate (:rhs this)))))

Then you are adding (and defining) the Evaluatable interface to a record BinaryPlus (which can be coming from a different library and already be implementing other interfaces)

BinaryPlus records can still be used where they were used previously, but you can also use them with the enhancement you added

Clojure's protocols are basically the dynamic language equivalent of type classes.

> I get the impression you'd need to explicitly create a new type class?

In the same way you need to use defprotocol in Clojure. You can implement existing type classes on types without defining a new type class.

Yep pretty much. It’s “open world” polymorphism.

> This example just scratches the surface.

It most certainly does. The point is to help people lay eyes on the shape of code, in context of a public concept.

The post I've submitted is the "Clojure-magic" deep-dive you want!

Greenspun's tenth rule:

> Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. > -- Philip Greenspun,

So. Much. This!

Circa 2013-15, I was a QA having to write some Selenium / WebDriver test suites for the web application side of our product.

At the time, when our team was tiny, Clojure was one of our "company languages" (besides JavaScript/YUI for web, and iOS/Objective-C, Android/Java for our mobile SDKs).

Our extant web testing framework was written in Clojure, so I continued refactoring that to support our people through "hypergrowth" feature churn including a web tech migration from YUI to React (took about four years for YUI to completely factor out, IIRC).

I did not fully grok just how much Clojure's multiple dispatch facility helped me then. I knew that feature was useful, of course, but the magnitude became clearer as the years progressed.

It allowed a single person---yours truly---to address the real needs of a ridiculously complex moving target (because B2B web products be like that --- full of special cases and customisation options and spooky effects at a distance).

I would not want to try supporting a test suite in that sort of situation, without at least these Clojurish programming facilities, that I have come to expect as a given these days.

Talk: https://www.youtube.com/watch?v=hwoLON80ZzA&list=PLG4-zNACPC...

PDF Deck (contains reading references): designing_object_functional_system_IN-Clojure_2016.pdf, here https://github.com/adityaathalye/slideware/

Incidentally, I implemented the refactored test suite API using the "PageObject" abstraction. An apt OO abstraction is worth its weight in gold, provided the implementation composes naturally.

Thanks, Martin Fowler! This very post helped me back then: https://martinfowler.com/bliki/PageObject.html

(edit: formatting, small bit of context)

Enjoyed this demonstration as well: https://max.computer/blog/solving-the-expression-problem-in-...

One of the drawbacks about protocols mentioned in the talk is no longer correct. Protocols can now dispatch on metadata, not just type.

    (defprotocol Dog
      :extend-via-metadata true
      (bark [_]))

Oh. Interesting. I had to look it up:

When :extend-via-metadata is true, values can extend protocols by adding metadata where keys are fully-qualified protocol function symbols and values are function implementations. Protocol implementations are checked first for direct definitions (defrecord, deftype, reify), then metadata definitions, then external extensions (extend, extend-type, extend-protocol).

(def component (with-meta {:name "db"} {`start (constantly "started")})) (start component) ;;=> "started"

Kind of a crazy feature.

Note that "full" solutions are possible in standard OO languages as well (without the usual visitor vs subclasses trade-off): https://news.ycombinator.com/item?id=45210114

I agree, given that (JVM) Clojure is literally just a Java Library.

From a programmer's point of view I don't want to have to write all the machinery that makes for a "full" solution (that is also correct and performant enough)...

In fact I can't! Which is why I hope more programming languages make that machinery available, because the "X/Y" problem (Type / Method) manifests itself (pretty easily) in every single web app out there (and likely most software ever written).

Reply on news.ycombinator.com