Hacker News Reader

Comment by sandGorgon | Hacker News Reader

Question - there was a recent post about how the dailymail group moved 140k lines of java code to 4k lines in clojure.

Since I have never programmed much in Java (I went from c++ to ruby!), could someone tell me where the gains are coming from ?

I know that eliminating the setters and getters (and the private/protected/etc system) would eliminate a lot of code, but I dont understand if it would be that much.

Possibly, I'm also trying to understand if they would have gotten similar improvements in kloc if they had switched to.. ruby/python for instance.

View on HN

Hi, I'm the CTO at the MailOnline so can comment. The biggest reduction in loc was due to the move from a relational DB to elastic search containing flatter representations of our content. This basically meant that we could delete thousands of lines of boilerplate and create a very simple processing pipeline to render content, and for this clojure really shines of course.

The REPL made the first week of exploring the problem and solution space really effective, and even as a node guy I was impressed with what could be done.

I can say that there is definitely a big productivity boost, but as others have said it isn't purely the language, but rather thinking about the problem in a completely new way that really did it. I do credit clojure though with giving our engineers the ability to think differently, as that is often the hardest part.

There was a comment I made in a speech recently about functional programming being frictionless for content management system design, this is what I was alluding to.

I don't want to start a flame war, but we built an earlier version in ruby (with one of the best OO guys I know leading it), and it was much more complex through attempting to model our home page as a set of inter related objects, using visitor patterns to traverse etc., than the clojure version.

very interesting. I had a similar experience with elasticsearch in modelling a ecommerce catalog - however that was with jruby. In fact, it took me quite some convincing to not go with something like mongodb - elasticsearch served us quite well.

Could you comment a bit more on the interaction of functional programming to content management ? Intuitively, it feels to me that a lot of it has to do with the NOSQL-like data model of elasticsearch. Being a lisp beginner, I feel that a visitor pattern is very analogous to "map" in functional programming - again, without all the OO getter/setter functions (e.g. "accept", "visit").

A lot of the simplification was definitely down to flattening the structure that was represented in 20-30 tables down into a smaller set of documents in ES. But the functional approach to then rendering these documents simplified it further.

A functional map is definitely analogous to a visitor pattern, but don't underestimate how much of a productivity boost arrives by being able to see all of the code in a small set of terse statements in one place, vs having a set of fairly abstract classes with accept/visit scattered all over the place.

We just found that the OO solution resulted in more code and it was more difficult to hold the resulting model in your head as the number of things that could be visited grew - we run a very big site.

The other big enabler vs a traditional CMS approach is that we have a very clear separation between the part of our stack that allows editors to create content, and the part that renders it (connected via messaging) - so we aren't mixing r/w domains and our rendering pipeline isn't burdened with the additional complexity of having to understand how content changes, it just renders what it is given as fast as it can.

Java (at least prior to Java 7), has some severe limitations in how code can be abstracted. As an example, it doesn't allow you to pass functions as arguments to other functions directly. As a result, the canonical way to handle situations where you would do this (such as passing a comparison operator to a sort function), is to declare an interface, define the function in a class that implements that interface (command pattern, etc.), and then instantiate an instance of the class and pass that. Closures are imitated by explicitly passing data to class constructors and storing it in member variables, and currying imitated through patterns like the onerous abstract factory pattern.

Additionally, the abstraction features Java does offer are implemented with some heavy language syntax, and often can't be generated at runtime or pre-compile time, at least not easily, and so there's a limit to the abstraction, even taking into account Java Generics. That's something where homoiconic languages like Clojure have a particular advantage.

As a result, abstracting things in these ways (which in other languages is incredibly common) gets to be painfully verbose. Often, things that would be beautifully abstracted in these ways are just left unabstracted (as in-line boilerplate) - the verbosity introduced makes the code less readable, rather than more, and so there's a stigma against "over-abstracted" code. The result is that code that does, or could, make use of these things takes exceedingly more code in Java than in a language like Clojure. Of course, conventions in Java of, e.g., giving each curly brace its own line aren't helping its case in line counts, either.

> Of course, conventions in Java of, e.g., giving each curly brace its own line aren't helping its case in line counts, either.

My comment is basically a free-association from this.

In C and Java both (and, um, CSS), the people I've encountered seem to strongly prefer giving each matched pair of curly braces one line (that is, the opening { doesn't get its own line, but the closing } does).

In lisps (to my knowledge) it's more normal to have all your closing )s stuck together at the end of the previous line. Using a clump of )))))s to discern exactly which scope you're in afterwards is tricky (I'm pretty sure that's exactly the "problem" that the C/Java style is trying to solve), so you look at the indentation instead.

Python formalized the indentation-is-scope paradigm, which means a malicious programmer can't trick you into believing you're looking at a different scope than you are, but my Python code always ends up being a lot taller than I feel it should be, so I have a lingering sense that Python took it a little too far. It turns out that I occasionally appreciate the option you get with lisps to put a short bit of code all on one line and mark the parse tree with parens instead of with line-breaks-plus-indentation.

(All that said, I do like Python a lot.)

Using a clump of )))))s to discern exactly which scope you're in afterwards is tricky (I'm pretty sure that's exactly the "problem" that the C/Java style is trying to solve), so you look at the indentation instead.

Another handy tool is an editor with rainbow delimiters. This feature changes the colour of each matched pair of parentheses so that they can be seen at a glance.

Seems to me that if you have a clump of )))), you should refactor that into individual functions instead. This would be akin to having a 500 line Java method.

There's really no excuse not to, since refactoring functional code tends to be as simple as taking a node in a tree and moving it out.

Since you use () for more or less everything in Lisp, from declaring functions or records(structs) to calling functions or doing comparisons (<, >, =, etc. are functions in Lisp), you will always get a clump of ) at the end of something.

In Java, if you have a class containing a function which itself contains an if-statement containing multiple lines. You will have three lines at the end of the file containing nothing but }. There is no reason you couldn't clump those together and thus end the last statement in your function with }}}.

The difference is, in Java it's good practise to give each } a seperate line. In Lisp, most people prefer to clump their ) together.

Also, you could easily end a function with alot of )))))).

(defn unique-large-squares [list-of-nums] (count (unique (filter #(> % 100) (map #(* % %) list-of-nums)))))

Of course, if the clump of ) are off-putting, you could always use ->

(defn unique-large-squares [list-of-nums] (-> list-of-nums (map #(* % %)) (filter #(> % 100)) (unique) (count)))

I imagine yogthos was referring to this, but the clump of `)))))` itself is only a symptom of the real problem: that you've got too much nested logic.

`)))))` at the end of a function doesn't matter. The function's over. Even if you needed to make a change that would break the clump up, you generally navigate it from the opening side.

But if it's at the end of a fat true-expression of an if-statement, you probably have to add a `;; x is false` comment to remind yourself what's going on by the time you get to the false-expression.

Just like in Python when you get to an `else:` yet you have to scroll up just to find the matching `if` because somebody decided to roll their own SOAP client on the true path.

Dunno. There's just a kind of a bad design smell to a language where one needs, or sorely tempted to use colors to discern the basic syntax.

Aside from the fact that if you're color blind then apparently you're stuck with all the )))))))s.

As we're already off-topic, I'll chip in with my view.

I personally hate the opening { stuck on the end of the line in C/Java. For me it goes on a line on its own pretty much every time.

I've lost count of the amount of times there's been a subtle bug introduced by incorrect scope as a result of a { being missed from the end of a line and not being noticed due to the code being formatted as if it was there.

With a { sat on its own, directly lining up with the }, that's pretty much impossible to miss.

I've always put it on the end of the line and found it easier to parse visually than when its on a separate line. To me that just seems like a waste of space.

How do you quickly check to make sure that all of your closing braces match up with the relevant opening brace, rather than, for example, an if statement that you forgot to put the { on?

Hopefully the formatting will work for this -

For example,

    if (isOK)
        for(thing in list
        {
            doSomething(thing);
        }

    }

Screams out to me at very first glance that I've missed a {,

whereas

    if (isOK)
        for(thing in list){
            doSomething(thing);
        }
    }

needs me to scan to the end of each conditional line to see where the problem is.

For me, it's the other way around.. the second instance is easier to spot because the visual cue of associating the block with the statement is more obvious if you assume beforehand that each statement has to end with the brace.

As for closing braces, I associate them with the start of the statement itself (the if and for) and not the opening brace, assuming the indentation is consistent.

I'm not arguing one is necessarily more right than the other - just that I prefer the way considered less right by consensus.

I get the impression that the { on the end of the line is the one that most people prefer, so I think you're more with consensus than me.

The argument about taking up less space, I get - although given the massive monitors that we all have these days, that seems far less of a consideration than when I used to work on 20-line terminals.

What I just genuinely struggle to understand is how it's clearer (and if I managed to understand, it might help me parse other peoples' code).

Let me turn my code into something that actually has a matching amount of braces

    while (readLine){
        if (isOK)
            loadSomething();
            for(thing in list){
                doSomething(thing);
        }
    }

versus

    while(readLine)
    {
        if (isOK)
            loadSomething();
            for(thing in list)
            {
                doSomething(thing);
        }
    }

If I was quickly parsing the first code, I'd assume everything was fine and that the first } was closing the if(ok). To spot the defect, I have to scroll up, find the matching control statement and check to the end of its line to see whether it had a {.

With second one, even the most rudamentary scan tells me that there's an opening and a closing brace that don't match up. All I have to do is scroll up and see whether there's a matching { at the same indentation.

I'm not saying you're wrong - if you are, most people who write coding articles, or code that I have to look at, are also wrong. But I would love to know what it is that other people can see easier/quicker in the first code than they can in the second in.

> I personally hate the opening { stuck on the end of the line in C/Java.

It's an abomination.

Some reduction in lines of code may have come if they had rewritten it in Java too, due to better understanding the problem space after having written the first version. Java has a fair amount of boilerplate type code other than getters and setters. For example forcing you to use "classes" for everything, even when you really want a procedural or functional API. Some people try to write semi functional style programs anyway using object oriented "patterns", which can sometimes lead to really just a lof more lines of code too. Also explicit type annotations take up space. And the Java style of using exceptions can be quite verbose. There may also be better designed libraries to do a lot of things for you automatically in clojure. Macros are very helpful too. But I'm just making all of this up, haven't actually seen the codebases to compare them.

Its very possible they would have gotten significant (though possibly not equivalent) reduction if they reprogrammed it directly in Java. An old codebase often has a lot of useless cruft and over-abstractions and so on, and you do better each time you code something. The experimental control is not the old code base, it is translating the same code base back to itself with similar engineering effort. No wonder few people actually try to gather that sort of data.

There is a lot of ceremony in Java. If I want a single polymorphic function I'll create an interface and a bunch of classes to implement it, instantiate those classes and pass them to where they need to be.

In a functional program, I'll just create the function and pass it around anonymously if needed. The function will in itself will likely be simpler than the Java equivalent and stateless.

There will also be less DTO type class code for carrying around dumb collections of data, replaced with simple records and lists.

I'm not at the stage where functional programming has really clicked for me, but after writing some Clojure and Haskell recently, the sheer amount of code you have to type in in Java to express yourself is really starting to bug me!

I think it's also in the mentality as in it's perfectly fine in lisp to use simple lists as data structure, whereas we mostly use wrapper objects around primitive or collections in Java.

For instance, you'll often see a "Person" class definition with getters, setters around the .adds collection function. The idea is to encapsulate the collections as much as possible so that if it changes down the road, nothing else is affected but that simple method. In lisp, however, you'd probably just (let (people '("tom" "john"))) and be done with it. I'm obviously exaggerating, but hopefully you get my point. It's similar in Python where I use dicts all the time, but feel bad using it in Java or C++ without abstracting or encapsulating it.

It's not so much a syntax issue (even though clojure syntax is less verbose than Java) but mostly the mentality behind the idioms IMHO.

Reply on news.ycombinator.com