Clojure: The JFDI language

I started up with clj and I am trying kinda heavy arbitrage here :)

Managing a transition like this is far from trivial for a company facing a variety of immediate demands and constraints, even if it promises to pay dividends in the long run.

The problem with this of course is that when a company tries to switch their more mundane engineers to Clojure, the project will likely fail.

Help?

The language, or its aficionado -- either way, specific technologies have made me, in my own estimation, multiple factors more productive. Yes, there's the learning curve, and understanding things that are not explicitly stated line by line throughout the code. But then, there's getting shit done.

Unfortunately, a lot of people, processes, and institutions get hung up at the sentence before last of that previous paragraph.

I also turned other people onto some useful technologies, such as pointing a consultant at some reporting software that my department didn't want to spring for but for which -- of course -- the outside consultant had budget. I heard from her a month or two later that it had greatly eased things and turned her into a sort of reporting superhero.

Throughout my career, I've seen people who, both on their own as well as through dint of better technology awareness and selection, have been multiple factors more productive than their coworkers.

And yes, often they are fighting the trend and significant inertia, rather than being embraced.

The line on slide 111, "Engineers 3-10x more productive, smaller teams", is probably in reference to...

Slide 16: "Code is typically 3-10x smaller when written in Clojure."

Slide 110: "3-10x less code. Limiting factor in programming speed is essential complexity of the problem, not typing."

I have to take assertions like that with a sizable chunk of salt. The alternative would mean that there's a gigantic arbitrage opportunity that's being missed by a bunch of highly motivated people.

That there is a "a gigantic arbitrage opportunity being missed" is essentially the thesis of the talk.

He states the reason why (Clojure is cloaked in 50 years of Lisp misconceptions) on slide 21: "Lisp is an old weird AI language no one uses anymore. It's so slow it requires a supercomputer. It's impossible to hire people who know how to use it. Functional programming has been proven impractical. Also, parentheses."

The body of the talk is designed to dispel these beliefs.

Oh, it's a perfectly plausible number. It increases the productivity of each developer by a factor of 3 to 10 and also increases the maintenance costs by 3^n_engineers or 10^n_engineers. This seems to be the tradeoff that startups make.

Lisp has a bad reputation compared to what it is. People think it's impractical, that Lispers are arrogant (due to a few vocal but influential people in the Lisp community) and that it's archaic or weird. I've addressed those perceptions, but they're out there. Clojure is, in fact, a pretty damn practical language.

Now, the arbitrage argument: in larger companies, the concerns aren't engineer productivity so much as appearances. For a middle-manager to launch a meaningful Clojure/Lisp initiative would be to put himself out there in a major way, and since programmer productivity is only one piece of a much larger system, it might not have that big an impact on the company as a whole. It might not be worth it. (Would Google's market cap go up 3-10x if it replaced Java with Clojure? Almost certainly not. Maybe the fair value would improve by 1.05x on fundamentals, which would be drowned out by month-to-month volatility.) On the other hand, startups generally can't afford (or don't think they can afford) to experiment with newer, quirkier languages. VC-funded startups tend (surprisingly?) to have stodgy closed-allocation work cultures and boring tech stacks because they're expected to allocate 100% of the "risk allowance" to one department: business-model risk. That's not to say that there aren't other kinds of companies that can (and, in some cases, do) use more innovative tools; but the two dominant cultural forces (VC startups and large corporations) in technology don't favor it.

For the large companies, I'll admit that there's an Amdahl's Law sort of effect at play. Let's say that you improve productivity of engineers by 3-10x. Does that "3x" the whole company? As I alluded to for Google, probably not. It makes something else the limiting factor. For most firms, using the wrong programming languages is nowhere near their top problem. To use Google as the example, it might deserve a 5% bump on market cap if it included Clojure (a small gain compared to monthly volatility, so small that no one could prove causality and take credit) on the white-list but the fair value would go up 50-200% (again, on fundamentals, as I can't predict investor opinion) if it implemented open allocation. Programming language innovation just isn't the top priority for most companies. But for an individual developer, the advantage conferred by a powerful language can be pretty serious.

EDIT: (yes, being a little cheeky because as we know the physical time programming isn't everything in building a website)

http://deliberate-software.com/intro-to-macros/

In fact, common practice in Clojure is to define a your macros in terms of a backing function:

    (defn foo-fn [body]
      (fancy stuff here that probably uses syntax-quote somewhere))

    (defmacro foo [& body]
      (foo-fn body))

The REPL made the first week of exploring the problem and solution space really effective, and even as a node guy I was impressed with what could be done.

I can say that there is definitely a big productivity boost, but as others have said it isn't purely the language, but rather thinking about the problem in a completely new way that really did it. I do credit clojure though with giving our engineers the ability to think differently, as that is often the hardest part.

There was a comment I made in a speech recently about functional programming being frictionless for content management system design, this is what I was alluding to.

I don't want to start a flame war, but we built an earlier version in ruby (with one of the best OO guys I know leading it), and it was much more complex through attempting to model our home page as a set of inter related objects, using visitor patterns to traverse etc., than the clojure version.

Additionally, the abstraction features Java does offer are implemented with some heavy language syntax, and often can't be generated at runtime or pre-compile time, at least not easily, and so there's a limit to the abstraction, even taking into account Java Generics. That's something where homoiconic languages like Clojure have a particular advantage.

As a result, abstracting things in these ways (which in other languages is incredibly common) gets to be painfully verbose. Often, things that would be beautifully abstracted in these ways are just left unabstracted (as in-line boilerplate) - the verbosity introduced makes the code less readable, rather than more, and so there's a stigma against "over-abstracted" code. The result is that code that does, or could, make use of these things takes exceedingly more code in Java than in a language like Clojure. Of course, conventions in Java of, e.g., giving each curly brace its own line aren't helping its case in line counts, either.

For instance, you'll often see a "Person" class definition with getters, setters around the .adds collection function. The idea is to encapsulate the collections as much as possible so that if it changes down the road, nothing else is affected but that simple method. In lisp, however, you'd probably just (let (people '("tom" "john"))) and be done with it. I'm obviously exaggerating, but hopefully you get my point. It's similar in Python where I use dicts all the time, but feel bad using it in Java or C++ without abstracting or encapsulating it.

It's not so much a syntax issue (even though clojure syntax is less verbose than Java) but mostly the mentality behind the idioms IMHO.

In a functional program, I'll just create the function and pass it around anonymously if needed. The function will in itself will likely be simpler than the Java equivalent and stateless.

There will also be less DTO type class code for carrying around dumb collections of data, replaced with simple records and lists.

I'm not at the stage where functional programming has really clicked for me, but after writing some Clojure and Haskell recently, the sheer amount of code you have to type in in Java to express yourself is really starting to bug me!

People want to believe the world is simple, and that learning one PL is enough. Once you know Java, why learn another one? Either they're all the same, or they're so radically different they must be weak in some critical way.

When you grok Lisp, and realize that it is actually practical, you realize that you've only scratched the surface of the PL problem... and that's scary for a lot of people, especially as there's a tendency to overestimate the barriers between languages. (The Internet runs on components written on all sorts of different languages.)

it's worth mentioning that the dot product would be normally be written as (reduce + (map xs ys))*

Yes, I had that thought, too.

In fact, that formulation is too slow for, say, large-scale ML. I've been getting a lot of mileage out of Prismatic's hiphip library... and I really look forward to, about 5 years from now, when Clojure is as mature for numeric programming and ML as Python is with Numpy, Scipy, Numba et al.

Can you recommend a good picolisp resource?

I'm looking for another language with similar philosophy to Clojure's, but with a better ecosystem.

Also, syntactically Scala is a lot more complicated than lisp. It's a bit closer to Java, but that doesn't make it simpler. Just more familiar.

When evaluating language tradeoffs the scala community values power more and the clojure community values simplicity more.

Scala wants to provide you with the most powerful language possible. Clojure wants to make it easy for you to mold your own language.

Scala language tooling works at compile-time and analyses the language statically. Clojure language tooling connects to a running environment and uses introspection. Scala has much better static checking and clojure has much better introspection. While both languages have a repl, the clojure community puts much more emphasis on live, interactive programming.

Clojures syntactic extension mechanisms are much simpler, being based on sexps. Scalas are powerful (eg can access the lexical environment) but much more heavy-weight.

Clojure has run-time eval. This makes it possible to do eg domain-specific jit (eg compiling DSLs -> clojure code -> java bytecode on the fly). Scala has a run-time interpreter which I haven't used but is reputed to be slow.

I haven't yet explored non-jvm backends for scala but I suspect that clojure will eventually dominate on that front for two reasons: 1) when clojure-in-clojure is finished porting clojure to a new platform will be far easier than scala and 2) clojure was designed from the beginning to be a parasitic language and delegates many features to the underlying platform.

Scala breaks backwards compatibility much more often than clojure. Whether this is a pro or con is up for debate. There are lots of small niggling things in clojure I would like to change but on the other hand maintaining scala code across compiler updates is a pain in the ass.

Personally, I believe that the number one problem a language needs to address is managing complexity. The clojure community largely seems to get that and is driving for simple, orthogonal language features and libraries to an extent that I've not seen in any other community (eg http://blog.getprismatic.com/blog/2012/10/1/prismatics-graph...). In addition, the most exciting research I've seen on that front (eg http://www.vpri.org/pdf/tr2011004_steps11.pdf http://boom.cs.berkeley.edu/ ) and most of my favorite tools rely on powerful introspection and runtime code generation. I think Yegge describes this better than I can - http://steve-yegge.blogspot.co.uk/2007/01/pinocchio-problem....

I would use Scala for a personal project where I felt I needed compiler-time typing: I was writing something for the ages, or where bugs might be truly intolerable (say, high-frequency trading) but code would remain small. I wouldn't recommend it for a huge project, though. (Then again, I generally wouldn't recommend huge projects.)

Scala tends to fall down on large projects, especially if the developers aren't skilled. Cyclical dependencies (yes, I know they shouldn't exist, but they will if you have a team of 25+ engineers and a wide skill spectrum) defeat incremental compilation, which matters when your compile speed is ~500LoC/second (Scala's compiler is very powerful). Also, SBT is a nightmare. I haven't used Scala's macro system but I can't imagine it being as clean as Clojure's.

I like Scala on its terms alone, even if I've seen it go wrong on large codebases. (Java has the same issues, but it has tools that rescue you. It's an ugly hackish thing, to rely on tools to deal with code messes; but they're there.) I like static typing (at least, the theory that comes from it.) However, I tend to think that what most programmers really want when they say they want "static typing" is something more like Prismatic's Schema: runtime checks that can be turned on or off (performance) as needed.

I think Clojure's easier to read and to write. It just looks harder, because people exaggerate the parentheses problem, which is not much of one at all. If anything, the only thing that's somewhat hard to explain to newcomers is quoting (e.g. why '(1 2 3) returns (1 2 3) at REPL but (1 2 3) is an error). The quotes were the hardest thing for me to get right when I started in Lisp.

I'm getting tired of statements like this. I've been using Clojure for a couple of years now. I got over the () "issue" after a couple of weeks, once I discovered Counterclockwise for Eclipse (and, eventually, Emacs paredit or smartparens). You just don't notice them after a short period.

In general, editors that "understand" Lisp-like syntax are more powerful than other code editors. Being able to select forms - anything enclosed in a (), {} or [] - with a key combination and move them easily around the hierarchy (also see barfage & slurpage in the paredit cheatsheet: http://www.emacswiki.org/emacs/PareditCheatsheet) makes editing code faster than other languages.

As for the "easy to read" argument, I now find Clojure easier to read than, say, Java because there's generally less of it.

however, i have to differ with you completely regarding scala being easier to read. as someone who regularly works with scala code i find it incredibly difficult to read in comparison to clojure. Lisp syntax is different from what most people are used to but is very simple; scala looks more like the syntax that people are used to, but is deceptively quite different, and is quite complex.

core.async has state (when buffers are used, in particular) but state is not exactly the same thing as mutability. Producers and consumers don't have unconstrained access to channels, in the way that users of a mutable cell (a Clojure atom, a ref cell, or just a variable in many languages) can clobber it at will.

Likewise (although I didn't get into this) state doesn't necessarily violate referential transparency. Memoization uses state as an optimization, but doesn't interfere with RT.

That might change. Clojure's been making some pretty impressive in-roads into the enterprise.

At this year's Conj, there were a fair number of companies hiring. Staples Innovation Labs is apparently hiring a lot of Clojure people.

What part of the world are you in?

Adjectives are properties or classes of things, like Numeric or Closeable (with-open pattern) or Monad. The Haskell solution is type classes. The Java solution is OOP. The Erlang solution (fairly similar) is the actor model, which is the inspiration for OOP's principle of locally interpreted functions/methods (as opposed to global function definitions). OCaml's is functors, which are extremely powerful but hard for most people to use.