The Python Paradox is now the Scala Paradox | Hacker News Reader

The Python Paradox is now the Scala Paradox

martin.kleppmann.com

118 pointshendler43 comments

This is from 2009. Scala has only improved since then, although it's still not a "mainstream" programming language by any stretch. You can talk to a crowded room of programmers who either haven't heard of Scala, or have heard of it only in passing and don't really know what it's about. That's fine.

Programmers who are interested in learning and growing will always be attracted to new languages and technologies. What's interesting about Scala is that it's not solely academic. You can learn and do useful things with it at the same time. Later, when you've learned more, you can go back and improve your old code for added beauty and performance. That's a fun process, I think.

I think the title could be updated for 2011 - "The Python Paradox is now the Functional Programming Paradox". Scala, Clojure, Haskell, and Erlang - all your points apply equally well to them.

The author says exactly that, near the end of the article. He names those languages in particular.

It's "whatever programming language is not mainstream paradox" to be precise (and the author of the blog post would agree since he also nominates Haskell, Erlang, and Clojure). Which is obvious. And since "whatever programming language is not mainstream paradox" is a mouthful, "Python paradox" is a much better name. Those unfamiliar with the term can always be directed to PG's article to enjoy.

EDIT: Using "esoteric" languages can also be a good long-term hiring strategy. It's pretty easy to get a place in people's minds as the "go to" company to work with a language. Google did that with Python, Twitter is doing it with Scala, Basho are doing it with Erlang.

Hmm my next project has to be a website written in brainfuck...

But more seriously I did write a search engine using only sed for a little site of mine (http://toybin.org/) just to see how far I could push sed. Its only feature is that you can have negative works like 'blue -car' will show search results for blue and not car. Haven't yet found any hard core sed hackers out there though... :D

Edit: GitHub would also be another Erlang company (with a ruby frontend)

> Using "esoteric" languages can also be a good long-term hiring strategy.

Actually, it's a good short term hiring strategy but a terrible long term strategy. Most of these languages remain a niche (Ruby, Scala, Clojure, etc...) and hiring more than a couple of people that know how to use them turns out to be a very, very tough challenge.

I have seen several companies try that and then revert back to Java just because they couldn't staff up to meet the demand.

Overall, color me still extremely skeptical of Paul Graham's original argument (which was that Lisp gave his company an advantage over the competition). Interestingly, ViaWeb was quickly switched back to a more mainstream language shortly after PG sold out. Deduce from that what you will.

I deduce from this that the fault was not in the language, but in the company, specifically in the resume filter that asks for 3+ years of production Clojure. A programmer who learns languages can learn Clojure for production work. That's better than trying to bring a soaring program back down to earth (or for C++, chaining it to a cart and dragging it into the salt mines).

But don't listen to me, here's Paul Graham from Beating the Averages (Footnote Edition). And ponder this question: why didn't Yahoo just rewrite the templates in C++?

[1] Viaweb at first had two parts: the editor, written in Lisp, which people used to build their sites, and the ordering system, written in C, which handled orders. The first version was mostly Lisp, because the ordering system was small. Later we added two more modules, an image generator written in C, and a back-office manager written mostly in Perl.

In January 2003, Yahoo released a new version of the editor written in C++ and Perl. It's hard to say whether the program is no longer written in Lisp, though, because to translate this program into C++ they literally had to write a Lisp interpreter: the source files of all the page-generating templates are still, as far as I know, Lisp code. (See Greenspun's Tenth Rule.)

I deduce that some people prefer some tools, and you get a big win when you match a highly skilled person with tools they feel best augment those skills. However, it's the pairing that is the big win. If I had to guess as to which element was more important, I'd bet that the skilled person with imperfect tools is better than a Blub programmer with tools that are above him.

PG left the equation, and when he did, a big argument in favor of keeping Lisp around did as well. Without another similarly skilled Lisp hacker to fill the void, it's easier to make a strong case for a more mainstream language so that ViaWeb could still maintain some forward velocity even if they never got to move quite as fast as they did when PG/Lisp were driving.

tl;dr - I bet that it's not Lisp that made ViaWeb tick - it was Lisp + PG.

> hiring more than a couple of people that know how to use > them turns out to be a very, very tough challenge

That's tackling the wrong challenge. Almost none of engineering staff at foursquare had used scala before starting here. But they're all talented engineers who can learn new skills and thought that learning scala would be interesting. We've had very few problems with people picking up the language.

That was my point: by choosing an esoteric language, you raise the bar for the people you need to hire, which makes hiring very challenging.

Now go scale that up to 50 or 100 engineers and you understand why it's not a viable strategy.

Try it yourself, go pitch a startup and then tell the VC's and their consultants that you're going to use Scala...

I know you're going to say that they don't know their stuff and you should be the one deciding, but it's actually the other way around: they know how to grow a business, and nobody with that knowledge will ever recommend to go with non mainstream language and technologies.

It almost sounds like you're saying raising the hiring bar is a bad thing.

got cut off on my phone; meant to add: why would you ever need 50 Clojure engineers?

For the exact same reason you might one day need 50 programmers in any language: if your company flourishes.

What makes Clojure so special that you'll never need that many?

1) raising the bar for the people you need to hire is a good thing. If you think that settling for mediocre engineers is ok, then you're doing it wrong

2) We've already have ~20 engineers writing scala code at foursquare. Not 50 yet, but the challenge of getting there will having nothing to do with our choice of language. If anything it's helped with hiring so far.

3) I did tell our VCs that we'll be using scala. They didn't have a problem with it. In fact a VC on our board agreed with me that "using cutting edge technology can give a real edge in recruiting."

Basically my experience growing the engineering team at foursquare has been the exact opposite of your claims.

Cross the bridge when you come to it. You only need a handful of programmers in the first couple of years of a startup's life. The better they are the better. Worrying about whether you'll be able to hire 20+ Clojure programmers is best left for when/if the company is successful.

The problem with esoteric languages in this modern era (with the web) is what I called the "college music problem" with the increase of college radio.

By the time the early 2000s hit, the people who listened to college radio no longer self-selected for people who knew a lot about music, but rather were just people who were generally anti-establishment. But they hadn't built up an especially rich musical knowledge base.

Fast forward to 2011 and languages. Now I'm almost as likely to see Clojure on a resume as C++. Yet the quality of applicants doesn't seem to have gone up all that much. TDD knowledge seems to be MUCH more prominent, but I haven't noticed an uptick in knowledge of algorithms, complexity, anything under the hood, or programming (as can be judged in a day long interview).

In 2011 the choice of language says less about the programming ability of the person, and probably more about their personality. Much like the college radio heads weren't especially musically sophisticated (on average, but of course some were brilliant), but they all seemed to share a counter-cultural vibe (even when you went from goth to hip hop to polka).

Do you think that choosing a non-mainstream language is really the result of some sort of counter-culture vibe? Some desire to be different?

I ask this because I am a fan of Python and Clojure. I got into both as a result of reading pg's essays on Python and Lisp many years ago. Both languages were sold to me not as an act of rebellion but as more-powerful, higher-level languages that could get the job done more quickly. I quickly found this to be the case, compared to Java which I was used to. Nowadays, I grimace when I have to use Java or C and find myself missing a language feature from Python or Clojure that could have saved me an awful lot of typing and repeating myself.

So, in my experience, was the choice of esoteric languages born of a desire to be different? No. Does it make me out as being an amazing programmer? Sadly no, not necessarily. I'd like to think, though, that it at least shows that I was enthusiastic enough to learn and use languages that I get a lot of benefit from.

Do you think that choosing a non-mainstream language is really the result of some sort of counter-culture vibe? Some desire to be different?

Not necessarily, but often so. More specifically, I think it used to signal someone who was more enthusiastic, passionate, engaged -- but now people will use them because they think it signals these things. And I don't think its intentional. But I think a lot like college radio, you see the guy who is leading the crew of outsiders and you see what he listens to. You see a lot of people who are signaling what resonates with you and follow that.

It's odd that C++ is one of the most denigrated languages on the planet -- yet it really does seem like the people I know who most write C++ code still are a lot more hardcore than those I know who do Clojure (and its odd that I may know more people that do Clojure than C++).

To put it another way the people doing the less mainstream languages are often putting themselves in the middle of it... "look at me -- look at the language I'm using". People doing C++ are saying, "look at my code -- look at my app".

You make a good point. The solution is quite simple though, look at what they did with the non-mainstream language and ask a few questions. In my case, it would take me a couple of minutes to find out if someone just read "Programming Erlang" or a few online tutorial or if they have indeed built an OTP app.

Upfront I want to be very clear that I'm not hating on any of the languages that he mentions in the article. okay, maybe secretly hoping that Ruby will go die in a fire somewhere, but let's put that aside.

What grieves me as I read that article is that it seems like he had a perfectly good way of doing it simply and easily, but then decided to go for a much more complex and risky solution (with what _he_ describes as poor tooling) ... for what? Simply to increase the difficulty level?

Take the analogy of a backyard pool. Instead of just running up to the pool and doing a bomb (or even a belly flop), he has to climb up on the roof of neighbours rickety garage, where he is going to attempt to do a triple twist half pike helicopter/superman maneuver in order to score higher from the judges. Problem is, he's got a real risk of either missing the pool entirely or cracking his head open on the concrete.

I'm not saying we should never use new languages or techniques or tools. What annoys me is that given the choice of doing something simple, or doing something complex, he chose the complex way, and then piled on the risk, with a side order of complexity.

From the article:

"Shortly after working this out and drawing my architecture diagram (a pretty insane-looking tangle of boxes and arrows on a sheet of paper)"

Shouldn't that have been a pretty big red flag? Sure he found some superstar programmer to pull it off for him, but I can't help but think that a little bit of darwinian natural selection would have been in order here.

The annoying thing is that you see this all the time in the Enterprise. Time and time again someone with architect in their title goes and makes an appalling horrible mess of the design, leaving the poor bastards at the coal face to sort it out and try to make the abominable crime against nature, reason and common sense actually work.

If you're dealing with message passing architectures a language like Scala is a good way to go. Actors or Mailboxes in F# parlance fit perfectly with the way the architecture was described.

His point about code fashion is also equally valid, what Seurat did with pointillism is impressive, printing a picture on on your inkjet is not, even though what Seurat did is essentially the same thing.

Coming from an imperative background writing in a functional style has risk, coming from a functional background imperative style has risk. You can implement the lambda calculus on a UTM and implement a UTM in the lambda calculus, for all intents and purposes they are the same, however, in terms of elegance they are far different. (And there are arguments on both sides as to which is more elegant).

I tend to agree with you particularly on 'enterprise architects' but the solutions are usually a reinvention of the wheel rather than marrying two disparate technologies that actually fit quite well together. Scala or another functional language is a great fit to a message bus architecture. Doing something for the difficulty of it usually doesn't attract a lot of smart minds, putting two things together that at first one wouldn't think of but that makes sense WILL pique the interest of a talented developer because he wants to see if it actually bears fruit.

I'm sorry, I think I don't understand the point you are trying to make. Are you really complaining that Java doesn't do messaging? Surely not, because that would be like saying that Lisp is okay, but it would be better with a heap more parenthesis.

When you get down to it, pretty much any language with even a rudimentary ability to manipulate strings and connect to a socket can do messaging. But right from day one Java has had the capability of communicating with remote machines baked in.

Oh wait, I just checked, I'm wrong. The Remote and Serializable interfaces date since Java 1.1, NOT since day 1. My bad. Of course 1.1 was released Feb 19, 1997 so RMI is almost exactly 14 years old at this point.

So my conclusion is that if you've been living under a rock for exactly 14 years, you have an excuse when you say that Java can't do messaging.

----

Someone always comes along and says "what about $blindingly_obvious_thing and gets massive amounts of karma for doing so, so let me just say that yes, I am aware of... bugger it, I'll do it myself and reap the karma. Muahahahaha!

...

...dang, there's no reply to my own message option. (For obvious reasons) Oh well, we'll have to settle for option #2 which is that someone comes along and posts an objection which has nothing to do with any of the posts before it, and gets masses of karma for it.

----

Addendum:

If you're implementing a messaging system in your handy dandy language with sockets and strings, and you want it to be robust, you will of course need to take into account this:

http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Comput...

or a more in-depth look at them:

http://www.rgoarchitects.com/Files/fallacies.pdf

cheerio, pip pip

Does anyone know of a concise way to learn Clojure or Scala? I'd like to learn one of these languages, but I don't want to wade through interminable books to do it. Rather, I'd prefer it if there were a place where I could quickly pick up some basics and get a few ideas/exercises I could work on to discover the language on my own, a bit like "Learn Python in 10 Minutes" (http://www.korokithakis.net/tutorials/python).

For Scala you could check First Steps to Scala(http://www.artima.com/scalazine/articles/steps.html).

For Clojure you have very nice video introductions by Rich Hickey on blip.tv.

Ah, thank you, that primer looks to be exactly what I wanted.

Scala can run in command line mode, which is pretty useful for learning.

If you are coming from Java, as I was, the learning curve was pretty shallow towards doing useful things. http://www.ibm.com/developerworks/java/library/j-scala01228....

Thank you, I come from Python but I should be able to make some sense out of it...

One of the hardest parts of getting into Scala for me (coming from mostly Java and Python) was that the yield keyword means something significantly different from what it means in Python. This actually broke my heart early on, as Python's generators are one of my favorite features from any language.

Delimited continuations in 2.8 make up for it and then some, but they're much harder to grok, imo. Python's yield gets you 80% of the same benefits with about 20% of the brain bending.

For Clojure, this is a good place to start: http://en.wikibooks.org/wiki/Learning_Clojure

I also like the lengthier, but most excellent, tutorial by Mark Volkmann: http://java.ociweb.com/mark/clojure/article.html

Project Euler worked well for me as warm-up exercises. And I just found a Wiki of Project Euler solutions in Clojure: http://clojure-euler.wikispaces.com/

Fantastic, thanks. Especially the project Euler idea, I was under the impression that they were Python-specific problems (in retrospect, I have no idea why I ever thought that).

You may be thinking of the Python Challenge; I could see them being mentioned together when someone asks for programming puzzle sites.

Probably, yes. It's one of those things you never realise are wrong until you think about them.

+1 Mark Volkmann's tutorial. That's what got me started on learning Clojure. Of course, once I'd gone through that, I still waded through this book and got a lot of value from it:

http://pragprog.com/titles/shcloj/programming-clojure (which was current at the time that I read it but might be a bit out of date by now since I think it covers Clojure 1.1)

Yep, I've saved that tutorial and am reading it, it looks comprehensive and on one page. I'll have a look at the book too, thanks.

The Tour of Scala is very succinct. It is basically a list of language features and how they manifest in Scala.

http://www.scala-lang.org/node/104

Each feature has both a description and an example. I became productive very quickly using almost exclusively this tour.

Try the book, "Seven Languages in Seven Weeks" which covers Clojure, Scala and a bunch of other languages. I've only just got it and it looks good so far.

http://www.amazon.com/Seven-Languages-Weeks-Programming-Prog...

Previously discussed on HN: http://news.ycombinator.com/item?id=831817

Thanks - hadn't seen this. HN didn't catch the dupe as the same article was submitted from a different url, which is now a 301 to this article.

> Of course Sam cannot live on technical stimulation and my gratitude alone

Alas. If only we could live on technical stimulation alone :(

"... Of course times have moved on, and Python (and Ruby, for that matter) are definitely entering the mainstream. They are still fine languages, but they no longer carry as much of an early adopter aura about them... what are the new technologies to look out for? ..."

http://nodejs.org/

http://news.ycombinator.com/item?id=2084732

(the last pardox thread)

I have been reading books about Scala and used it for scripting here and there. I gained some Scala experience over the last years, yet it still amazes me how often people state:

"Scala is not ready yet, but when it is, it is gonna take over Java as the next big language".

Years have passed and this has failed to materialize. Maybe it will never be 'ready enough' to take over? What do you think?

scala is quite complex. it has O(m + n) complexity from functional and "object oriented" (single dispatch, backported type system) requirements, plus whatever their interactions cost, plus the novel shit. I admire anyone who really grasps it in it's entirety.

as a mere mortal I gravitate to simpler things.

Reply on news.ycombinator.com