Hacker News Reader

> 1) All data flows through your system on separate tracks. Foo’s come in from foo endpoints and go to the foo database, and very rarely do data sets cross paths. Differences in business logic can be separated by API endpoint, kafka topic, or some other difference that lets you separate the call paths thoroughly.

My instinct is that you could solve general "where does this come from" and "where does this go" questions with metadata. I believe that was one of the driving reasons for metadata in the first place.

> 2) Every type of data in your system looks different, so that you can easily determine whether or not a given piece of data is a foo or a bar once and send it down the right call path in one place.

Clojure spec, malli, schema are libraries that deal with this kind of problem. They are very expressive.

> 3) In any case where similar types of data must be treated differently, it’s possible to organize your code in such a way that you only have to build up the cond-tree once, and you can use different call paths to treat the data differently.

A combination of spec (or others) and multimethods (arbitrary dynamic dispatch) would come to mind here as a solution.

I assume these things are known and were discussed. Are there specific trade-offs that didn't work out?

It's kind of interesting that you're responding as if all of those three points are something against Clojure, when in fact they were cases in which Clojure would work really well. I made that clear.

> I assume these things are known and were discussed. Are there specific trade-offs that didn't work out?

As is always the case when a team switches programming language, the problem I described was the straw that broke the camels back for us. But fundamentally the issue in this case was "it sure as heck looks like we're trying to approximate class based dispatch inside of Clojure, this sure as heck is silly. Let's just use a language that actually wants to be used this way".

Makes sense. I approached this from a curious point of view, not from a combative one, to get a sense of where the problems were.

The general problem you describe was also mentioned in the podcast. Trying to write like language X in language Y. This can also be observed within the Go community; people writing Java or JS style programs in Go, especially early on.

> We ended up in situations where we had lots of data that looked very similar entering from common endpoints that required different business logic at multiple points in the pipeline. If you’re writing idiomatic Clojure this is the toughest case possible, as you end up littering your code with extremely similar cond-trees, which makes extension and verification unnecessarily hard.

Isn't this where you would use multi-methods? If you can determine what the "type" of data is via `cond` (sounds like home-grown pattern matching?), couldn't you replace these situations with multi-methods that switch on the same conditions?

Multi-methods suffer the same drawback as repeated conds. You end up having to maintain multiple multi-methods in multiple locations that all need to cover all the same cases. If anything else, multi-methods are marginally worse than conds, because the actual implementation can be spread out and harder to visually check.

If you can get away with one multi-method, that's fine. The moment you start needing multiple multi-methods in different points in your pipeline then things begin to suck.

> sounds like home-grown pattern matching

Yup! And if you find yourself constantly writing pattern matches in Clojure, then you probably actually want classes, since that's a much better way to bind both data and behavior together.

Have you considered core.match ? I’ve used it to dispatch API calls based on just data shape.

I know that this isn't your intent, but suggesting that a bunch of professional Clojure programmers just didn't think to use core.match is borderline insulting. The problem wasn't "doing the matching is hard", the problem was "doing the matching in multiple places and making sure that we cover all cases in all places is starting to look like a maintenance nightmare".

> "doing the matching in multiple places and making sure that we cover all cases in all places is starting to look like a maintenance nightmare"

That sounds more like an indictment of functional programming rather than Clojure specifically. I’m not a seasoned functional programmer but I often read glowing praise from others about the use of pattern matching in their code.

I’m also not a professional Clojure programmer, but I thought that multi-methods were supposed to be the solution to the expression problem [0] that both class-based inheritance and pattern matching suffer from.

It was my understanding that multi-methods are supposed to allow open extension of behaviour without having to do what you just described; track down every instance, and cover every case.

In the case of multi-methods, couldn’t you just co-locate the method implementations next to the data they are supposed to operate on? You should be able to add/modify behaviours without worrying about what the other data types are doing.

Honest questions here. Just trying to learn more about the real world pros/cons of these approaches.

[0] http://wiki.c2.com/?ExpressionProblem

A good post on the Expression Problem, and the benefits of Clojure's solution to it: https://eli.thegreenplace.net/2016/the-expression-problem-an...

Also, I agree, it's interesting to hear his experience report.

The best summation I could possibly give is from the article you linked:

"In object-oriented languages, it's easy to add new types but difficult to add new operations. Whereas in functional languages, it's easy to add new operations but difficult to add new types".

Put more concretely: if your Java code base is full of one-off classes with little to no sub-classing or multiple implementations of internal interfaces, then Clojure might work for you. But if you have a lot of sub-classes and repeatedly implemented domain specific interfaces I would hesitate mightily before considering Clojure.

My team's domain was such that the operations were very fixed, but the types expanded constantly. There is only a fixed set of things that the back office can possibly do with a bond or a future, but new types of instruments come into existence (or sometimes our awareness) with pretty surprising regularity.

But the article you linked had a slight of hand trick with how they described multi-methods. They only give an example one multi-method. What if you need to emulate a Java interface with M methods on it? Well now you'll need M multi-methods with an implementation for each virtual "class". You quickly see how that falls apart if you need to add a type or an operation, since either way requires you to manually check and make sure that all operations are implemented for all types. Again, it works, but it's error prone.

Defrecord & defprotocol work much better. We ended up trying them and they were ... okay. They work, with some footguns, most of which I can't remember anymore, but if you end up going too far down that road (like we did) you end up asking yourself why you're writing Java in Clojure instead of just writing Java in Java.

> That sounds more like an indictment of functional programming rather than Clojure specifically.

I'd say that it's a problem (not indictment!) of Lisp style[0] functional programming, where there's a pretty hard wall between how you organize your data and your functions[1]. If we'd been using Haskell, we could have at least declared a type class for our shared behavior and used that to handle the different parts of behavior in the pipeline.

I say problem with some reservations, because your mileage will vary depending on the domain in question. My current team could use Clojure quite effectively since the domain is much more amenable to how Clojure approaches these problems. My position is less "this doesn't work" and much more "this has some tradebacks you should be aware of".

> I’m not a seasoned functional programmer but I often read glowing praise from others about the use of pattern matching in their code.

Pattern matching is superior to if/else/else if trees, full stop. If you have to have a bunch of if/else trees, you'd rather use pattern matching.

The problem comes when you start repeating the same if/else conditions (not behavior) in multiple places. In a language like Java, you would naturally start wondering if you should make a class or interface to factor this out somehow. In a language like Clojure you don't have nearly as many tools laying around to collect common behavior into shared locations. You can do it, but you'll be happier the less you have to use multi-methods and defrecords.

> I thought that multi-methods were supposed to be the solution to the expression problem

Multi methods are ... ok. There are a few footguns laying around with them, since using them suddenly makes imports side-effectful, but I can and have used them successfully.

The issue is that multi-methods inherently give you one function, and there's no real way to tie several of them together at all. That's great if your domain is organized in such a way that you can use a single multi-method in one spot to break up the dispatch tree. It's less great if you need to use multi-methods multiple times during processing to specialize based on similar traits of the data. You can do it, but it's error prone and you'll end up with a nagging feeling that you're just making a really bad object system using multi-methods.

> It was my understanding that multi-methods are supposed to allow open extension of behaviour without having to do what you just described; track down every instance, and cover every case.

Yes, and it's this open-ended nature that kind of screws you in some cases. If I have four multi-methods that all need to cover the same cases (with different behavior), it's real easy for me to forget one when modifying my code. That same flexibility in adding instances means that there's nothing preventing me from forgetting one too.

> couldn’t you just co-locate the method implementations next to the data they are supposed to operate on

Well, the data came from Bloomberg, so there was nowhere we could put them that would be "next to" the data. We could put them all in one namespace, but that starts getting unwieldy fast. We could separate them out into different namespaces for clarity, but now it's even easier to miss that you've forgotten things. It's a tough trade-off.

Oh, and multi-methods really messed with our dev tools at the time. Hopefully they've fixed it since then, but a lot of the time Cursive Clojure wouldn't refresh them properly, resulting in dozens of REPL restarts for those of us that preferred Cursive. That got old fast.

0 - As long as you pretend CLOS isn't a thing. This isn't a big deal, since lots of people like to pretend that CLOS isn't a thing.

1 - Yes yes, I know "functions are data too", that indeed was a neat trick ... half a century ago. In Lisps the hard wall is in organization; there typically is very little binding the data to functions like you get with objects[0] or even Haskell style typeclasses. Getting the right data into the right function is entirely driven by how functions are called, rather than by the data itself.

You've actually hit the real crux. Beyond the functional, beyond the Lisp, what is special and unique to Clojure is its style of data modeling. Using plain immutable data-structures creates a very unique style, and I think there isn't enough conversation about that. So it lacks best practices, and possibly support around it as well, in tooling and in style.

It's hard to discuss your particular case, but what I wonder is if there wasn't a different way to model things that would fit in the Clojure data-droven model. And if not, would there be new patterns or even some extension constructs to it that could allow it to fit?

Did you ever consider using Clojure spec or one of the alternatives before it? Runtime data shape/type inference is quite capable.

Spec was released after we switched to Java.

So just add some triage middleware (maybe Java based) between the data sources and your Clojure code, which separates the coarse-grained data into more granular flows for consumption by your Clojure code.

You can do that, but why? There's a dark irony in using a language that's propped up as a better alternative to Java, and then dealing with the pain to wire in Java halfway through the pipeline to overcome the perceived deficiencies of the language that was supposed to be a better alternative to Java. At a certain point, just using Java is much easier.