Horses for courses: choosing Scala or Clojure
rrees.me
rrees.me
But Rich Hickey is such an amazing guy, steering Clojure, with pragmatic solutions catering to developers while Scala often feels like an exercise in academic language development.
We can argue about the trade offs he made, I think Scala represents a reasonable point in the language design space: its not for everyone, but a lot of people like it a lot; it is probably more popular than Clojure (though LISP'ers tend to be more vocal in general).
It is definitely not a programming language for language enthusiasts, who tend to be more picky than programmers and developers and usually have their personal esoteric favorites.
Case in point: The lacking Scala API with a high percentage of method calls not described, not explained and rarely any descriptions of classes. Second case in point: In 2013 Scala is still abysmal slow (several seconds for some classes) when compiling (with SBT compile/JRebel reload or with Play compile/reload).
PS: Side note: Not to sound arrogant, but for the perception and spread of the language it is not important what Martin thinks, but in customer development style what my perception as the CTO of a company thinks that might decide between Scala and Clojure.
Part of the problem you are probably hitting is Scala's attempt to be statically typed while also being flexible, not really its attempt at marrying OO and FP; this tends to require lots of advanced typing concepts that many people are not yet comfortable with, and perhaps they never will be. Dynamic languages save a lot by pooing away the static type system; but of course there is then a lot they can't provide. I am a big critique of this also, we've had our disagreements, but really...at the end of the day it was a design decision made that is no less valid than other design decisions that could have been made.
So don't use Scala if it doesn't suit you, this is reasonable, your criticisms are reasonable, but I don't think it is fair at all to say that the designer was stupid or somehow misguided. There are more CTOs who are misguided than successful language designers.
By splitting out those concepts, you get a very solid foundation on which you can build pretty much any other paradigm.
Is it widely accepted that Clojure is better for concurrency? [1] I haven't used Clojure much, so I can't comment on their relative merits. But, I've played with actors in Scala some, and done a bit with STM, and they make concurrency relatively straightforward. It's plausible that Clojure has a better concurrency story (and this blog post has inspired me to try it out) but, concurrency is easy enough in Scala that I don't think of it as a weakness of the language, the way I would with something like C.
[1] This is not a rhetorical question.
I don't know Scala that well, but actors in scala still shares state, and doesn't give the same level of protection when it comes to changing state. Actors also introduce some overhead in case you just want to read a single variable from a datastructure safely. You have to send a command and receive a message, wheras in clojure you just retrieve the value you want. Actors however are, to my knowledge, easier to scale across machines as actors doesn't necessarily have to be in the same process, or even on the same machine.
I wasn't aware Scala had incorporated Clojuires STM, I thought Scala was married to Actors/Akka?
Also your right Clojure can use Akka just like scala so neither is any better for it, but scala can call native clojure code too. What I have found Awesome is that I can work on a scala project or library and hand it off to my co worker who is a clojure guy and he can incorporate it.
> Actors also introduce some overhead in case you just want to read a single variable from a datastructure safely.
... I think the large part which you are missing is that in Scala, everything is a library, there are no “privileged” operations or data structures:
In Scala, you have a huge set of options available which can be used where appropriate. There was a talk by the creators of Akka recently, and the most important message of their talk was “pick your poison”.
It's not like in Go where people claim that channels and “goroutines” are the solution to every problem or how the Clojure community treats STM/Agents.
In a sense, this also shows the maturity of Scala's developers and ecosystem, because they don't claim that X or Y is the ultimate solution to everything.
- Scala has a full range of immutable & persistent as well as mutable & concurrent data structures available. In general, they are faster and more efficient than the ones in Clojure, due to the heavy tuning they received and the fact that you can pick the implementation which has the best performance characteristics for your use-case.
- The collection library supports data-parallel operations since 2.9, but I think it's great that Clojure tries to catch up!
- Additionally, Scala offers a standardized Future&Promises implementation, as well as support for async blocks (also a libraray feature) in the upcoming version.
- On top of that, you can use java.util.concurrent or any other Java-based concurrency package without any impedance mismatch, like Netty.
- If necessary, you can also drop down to things like Threads, Locks, Queues, synchronization, @volatile variables or sun.mis.Unsafe.
- Actors offer solutions for higher-level concerns of concurrency, like distributed computation, failure tolerance and communication over unreliable networks. (I think the best question whether you need Actors is: “Is there a chance that message X never reaches the receiver?” If the answer is no, there is pretty much no point in using actors.) (There are currently some improvements under-way to make sure that no state can be accidentally captured in Actors, Futures, etc.)
- In addition to that you also have things like the Akka Dataflow libraries, Rx-style approaches or functional reactive programming.
- You can also use the STM library (or agents, ...) if it is the best tool for the job (most concurrency people in Scala say that STM is conceptually a dead-end though).
I hope the explanation helps a bit!
Well... Go offers mutexes (and other synchronizing constructs) becouse Go admits that goroutines and channels, while great for alot of problems, doesn't solve everything. Rich Hickey also just announced a new library bringing "goroutines" and channels to Clojure. Both Go and Clojure communities know that there are certain problems not suitable for their prefered concurrency tools, but that doesn't mean that new functionality can't be added as libraries, as in Scala.
> Scala has a full range of immutable & persistent as well as mutable & concurrent data structures available. In general, they are faster and more efficient than the ones in Clojure, due to the heavy tuning they received and the fact that you can pick the implementation which has the best performance characteristics for your use-case
You can still pick other datastructures specialized for your usecase in Clojure, or create your own persistant or mutable datastructures. All collections from Java, including arrays, are also available, and if these fits better than Clojure's default collections then you are free to use them. The default and easiest option is still to use Clojure's persistant collections as well as atoms, refs and agents for changing state. Since using mutable collections and datastructures requires a little more syntax, most developers will use the persistant collections and the concurrency primitives provided by Clojure. This in turn makes it easier to refactor concurreny into your code, as you normally don't have to change mutable collections to their persistant alternative, and you don't have to check if the procedural code you may have written is thread-safe.
On the flip side, Scala makes it easier to write performant code in a single-threaded setting, mostly becouse working with mutable datastructures and/or procedural code in Clojure isn't really natural, or shall we say "The Clojure Way".
> - The collection library supports data-parallel operations since 2.9, but I think it's great that Clojure tries to catch up!
Well, Clojure has had data-parallel operations bultin long before the new reducers library was available (pmap amongst others).
> - Additionally, Scala offers a standardized Future&Promises implementation, as well as support for async blocks (also a libraray feature) in the upcoming version.
Clojure has had Futures&Promises since the beginning, and asynchrounous code (go-blocks and channels) is available in the upcomming clojure.async library.
> - On top of that, you can use java.util.concurrent or any other Java-based concurrency package without any impedance mismatch, like Netty.
True, using java libraries causes a little impedance mismatch, but is still easy to do. Using stuff from java.util.concurrent is even encouraged when the builtin primitives just won't do.
> - If necessary, you can also drop down to things like Threads, Locks, Queues, synchronization, @volatile variables or sun.mis.Unsafe.
Same in Clojure
> - Actors offer solutions for higher-level concerns of concurrency, like distributed computation, failure tolerance and communication over unreliable networks. (I think the best question whether you need Actors is: “Is there a chance that message X never reaches the receiver?” If the answer is no, there is pretty much no point in using actors.) (There are currently some improvements under-way to make sure that no state can be accidentally captured in Actors, Futures, etc.)
"Pretty much no point in using actors" is IMHO an overstatement, but to each his own. What you say regarding improvements underway to prevent capturing external state is very interesting, and should be a great improvement for safe concurrent operations.
I'd argue that Scala supports a far more mature concurrency concept (message passing) as opposed to Clojure's STM. Refactoring Scala's concurrency features to interoperate with plain Java is much easier than refactoring a largish STM transaction.
Rather, I imagine that in such discussions, one should first focus more on immutable values; then the time model of identities and values (which encompasses the ref types, of which STM is just one); then things like pmap, reducers and stuff you get from Java.
[Disclaimer: I know next to nothing about Scala, and would surely enjoy playing with Scala's Akka within Clojure.]
When evaluating language tradeoffs the scala community values power more and the clojure community values simplicity more.
Scala wants to provide you with the most powerful language possible. Clojure wants to make it easy for you to mold your own language.
Scala language tooling works at compile-time and analyses the language statically. Clojure language tooling connects to a running environment and uses introspection. Scala has much better static checking and clojure has much better introspection. While both languages have a repl, the clojure community puts much more emphasis on live, interactive programming.
Clojures syntactic extension mechanisms are much simpler, being based on sexps. Scalas are powerful (eg can access the lexical environment) but much more heavy-weight.
Clojure has run-time eval. This makes it possible to do eg domain-specific jit (eg compiling DSLs -> clojure code -> java bytecode on the fly). Scala has a run-time interpreter which I haven't used but is reputed to be slow.
I haven't yet explored non-jvm backends for scala but I suspect that clojure will eventually dominate on that front for two reasons: 1) when clojure-in-clojure is finished porting clojure to a new platform will be far easier than scala and 2) clojure was designed from the beginning to be a parasitic language and delegates many features to the underlying platform.
Scala breaks backwards compatibility much more often than clojure. Whether this is a pro or con is up for debate. There are lots of small niggling things in clojure I would like to change but on the other hand maintaining scala code across compiler updates is a pain in the ass.
Personally, I believe that the number one problem a language needs to address is managing complexity. The clojure community largely seems to get that and is driving for simple, orthogonal language features and libraries to an extent that I've not seen in any other community (eg http://blog.getprismatic.com/blog/2012/10/1/prismatics-graph...). In addition, the most exciting research I've seen on that front (eg http://www.vpri.org/pdf/tr2011004_steps11.pdf http://boom.cs.berkeley.edu/ ) and most of my favorite tools rely on powerful introspection and runtime code generation. I think Yegge describes this better than I can - http://steve-yegge.blogspot.co.uk/2007/01/pinocchio-problem....
Q: What does java.lang.NoSuchMethodError:
clojure.lang.RestFn.<init>(I)V mean?
A: It means you have some code that was AOT
(ahead-of-time) compiled with a different
version of Clojure than the one you're
currently using. If it persists after
running lein clean then it is a problem
with your dependencies. Note that for
your own project that AOT compilation in
Clojure is much less important than it
is in other languages. There are a few
language-level features that must be
AOT-compiled to work, generally for
Java interop. If you are not using any
of these features, you should not
AOT-compile your project if other
projects may depend upon it.
“““
I'm unhappy with Scala's compatibility guarantees,
therefore I'll pick a language which
a) does not provide any binary compatibility guarantees
at all,
b) has no tool to figure out whether incompatible
changes have actually happened,
c) lacks any sort of guidance to figure out if library
A is compatible with library B short of downloading
and executing them together,
d) provides no clean way to cross-compile libraries
against multiple versions.
But hey, let's bash Scala for caring about these issues
and deride them for shipping actual, working
solutions addressing those issues.
”””
- Source compatibility between major versions
- Source and binary compatibility (both backward and
forward) for all minor and update releases of that
major version
Hypocrisy and cognitive dissonance FTW, I guess.
With regards to binary compatibility, what I think you are missing is that clojure libraries are almost always distributed as source code. The clojure compiler is fast enough to just compile everything at runtime (during development at least) and clojure has not broken source / api compatibility since the first major version. Where AOT compilation is useful is:
a) Improving application startup time
b) Shipping libraries which generate java classes at compile-time (usually to
c) Shipping proprietary libraries without source code
I would definitely like to see the community figure out binary compatibility between libraries to improve cases b and c but it's really not a pressing concern.
EDIT: On AOT compilation http://dev.clojure.org/pages/viewpage.action?pageId=1572898
I said "modern language".
> 2.
First, the changes are too large, then the changes are too small? What now?
It's bad if new Scala versions don't ship with full-scale library support, but it's also bad if developers start testing their libraries early?
I guess it is impossible to make you happy.
(as for spring, I'm working on a type-level alternative as we speak)
Not on an enterprise context, like it or not, most consulting projects on the enterprise world require interaction with them.
Multi-core concurrency means that your concurrent operations can happen at the same time and thus corrupt shared state.
Static languages are also usually faster, so I guess the tradeoff stands between raw performance and code size.
There are numerous succesfull projects using dynamic languages. Github, as far as I know, uses Ruby.
Wrong.
> ... and to ensure this type safety, they need more information from you, the developer.
Wrong.
Anyway, correlation != causation. If I would claim that there are some projects which are successful, despite being untyped, I could pretty much use the same data points to make my claim.
What? Static languages doesn't ensure type safety? Or errors due to wrong use of types doesn't get uncovered in unit tests? Unit tests runs the code, so when it runs code where wrong types or names are a problem, the code will fail, thus uncovered... Doesn't help for code you don't have tests for though, which instead require manual testing.
The type system does need more information from the user to allow the same kind of flexibility in a type safe environment. Polymorphism either through interfaces or class inheritance is basically a way to tell the compiler that "Hey, i know these objects look different, but they do support the function/method I'm calling, so relax ok?".
When someone says they don't understand the use of dynamically typed languages in a real world setting, it makes sense to mention that a company like GitHub mainly uses a dynamically typed language in their product. Heck, even Facebook uses a dynamically typed language quite extensively in PHP. Yes, you could make the argument that alot of successfull projects use statically typed languages, and naturally they do. The point is that dynamically typed languages work just fine in a real world setting.
I disagree with your notion of equating types with tests. They are fundamentally different. Types provide substantially stronger guarantees without having to execute any code.
> The type system does need more information from the user to allow the same kind of flexibility in a type safe environment.
Did you have a look at Haskell? Or any other language with type inference? The connection you are trying to establish between "untyped == no need to specify types explicitly" and "typed == necessary to specify types explicitly" is patently wrong.
Having a helpful compiler doesn't mean that the language has to suck. I don't want to offend you, but your comments create the impression that you are lacking experience with a modern, statically typed language.
I'm not equating types with tests. I'm saying that type errors comes to light when a unit test for a given code is run, because you will get unexpected results and/or an exception is thrown.
> Did you have a look at Haskell? Or any other language with type inference? The connection you are trying to establish between "untyped == no need to specify types explicitly" and "typed == necessary to specify types explicitly" is patently wrong.
I started learning Haskell but stopped when I came to monads. I've also experience with other languages that provide type inference like Go and F#. Go requires you to define interfaces (something you don't need in a dynamic language like, say, javascript). Haskell also has features that require the developer declare something upfront (types and type derivation), as do F# (discriminated unions and/or types).
My actual example was regarding class inheritance and implementing interfaces. Polymorphisism through inheritance or interfaces is great in a statically typed language, but not required or even possible in certain dynamic languages.
> Having a helpful compiler doesn't mean that the language has to suck. I don't want to offend you, but your comments create the impression that you are lacking experience with a modern, statically typed language.
I never said statically typed languages suck. I've developed an app in C# which was a great experience, and at my day job I work in C++ (which IMHO is a sucky language, but not due to being statically typed). Haskell is also a great language, as is Go and C#.
What I did say is that statically typed languages often require more code, which (depending on who you talk to) slows down development compared to a dynamically typed language. Statically typed languages does however usually entail better performance and (sometimes) smaller memory footprint than dynamically typed languages. There is a tradeoff, but that doesn't make one more suited for real world applications than the other, which there exists numerous examples of.
Generally not. Type inference does that for you.
I agree completely when it comes to refactoring/modifying code though. This can become cumbersome, especially if you don't have unit tests.
I don't see why there are static vs dynamic flamewars. From my experience, dynamic languages are good at doing more with less code, and getting somewhere faster. Stuff written in static languages tends to be more efficient and reliable. I'm happily using both.
And the line between static and dynamic languages is a bit blurry anyway.
In a functional language you are mostly passing around fairly pure concepts of data as lists and maps, and transforming and combining data between functions. This data is standalone, and serialisable, and so easy to understand; a REPL-oriented environment means that code gets implemented incrementally, building from some inputs to the required result interactively, so everything gets some testing as it is developed.
The static types of this pure data probably wouldn't be very interesting.
Compare with SQL - SQL lets you join and transform arbitrary tables using queries. SQL doesn't require you to create up-front static types to accommodate the shapes of the resultsets of queries that are include joins. The Clojure Way has a lot in common with SQL, I find.
[1] of course there are plenty of large real world systems written in Python etc.
You realize that DML/DQL is only one part of SQL? It also includes a DDL. All the types are right in there.
> resultsets of queries that are include joins.
I have never had to use DDL to declare the types of the results of a join. What database do you use where you do?
Static types != manifest types.
I was suggesting that the immutable part of FP (which is most of it) has a lot of similarity with relational querying, which is very successful in the real world.
Clojure is entirely different. When you write (f x) you know exactly how f is going to dispatch by looking what f is in f's namespace: either it is a simple function, a protocol method implemented by x (just like interfaces in Java) with the protocol defined in f's namespace, or a multimethod in f's namespace.
So in Clojure, you don't have dynamic dispatch at all other than Java-like polymorphism or multimethods whose implementations are confined to a single namespace.
[Disclaimer: I have never used core.typed.]
Can a Clojure expert comment on whether the Clojure part is as wrong/inaccurate as the Scala part?