A farewell note to a programming language
matthiasnehlsen.com
matthiasnehlsen.com
- algebraic data types
- structural recursion
- fold, map, and flatMap
- type classes
That, in my experience, will cover the majority of code you need to write in Scala and give a solid basis for a robust and comprehendible code base.
It's unfortunate that in Scala algebraic data types and type classes require a bit more code than in most other functional languages.
E.g.
sealed trait Foo
final case class Bar(...) extends Foo
final case class Baz(...) extends Foo
data Foo = Bar ... | Baz ...
These patterns are really truly shockingly simple to use. We teach them in introductory courses and the majority of people get them.
It's true that Scala has some warts, but as software engineers we ought to make decisions based not on our personal biases but on a consideration of the problem domain and tradeoffs involved. If you want 1) static typing, 2) a modern language, and 3) JVM compatibility then Scala is the best choice at the moment IMO. Remove one of those restrictions and other choices come into the picture.
Update: If you're interested in learning more about these patterns, on Friday we're going to send a free excerpt of our "Essential Scala" book to our mailing list. The excerpt will include material on algebraic data types and structural recursion. You can sign up here: http://underscore.io/newsletter.html
It can't run on the JVM, but to be honest, I've never actually observed in practice what's so great about the JVM (if you're running functional code). Everyone I hear claims that the JVM is great (left & right), but I just don't see it. Maybe I'm just ignorant.
This is particularly the case in the big data space, where all the major frameworks are run on the JVM.
For many people, interfacing with legacy Java code is also important.
And yes, I advocate using Scala much like Haskell.
(There is also the performance of the JVM, which is pretty good, but this is only a point worth mentioning when comparing to languages like Ruby and Python. Most functional languages have good compilers and runtimes.)
Don't get me wrong... it's good but not great (in my conception of things). Even if you find some indispensable JVM-based library, then why not just shell out to an executable jar (or whatever) that does what you need? At least you'll avoid classpath-hell that way.
I think the language that hasn't the biggest chance of upsetting the JVM is Rust. Mozilla talk about Rust as a C/C++ replacement, but I actually expect they'll see more interest from JVM focused developers. A lot of current systems programming uses the JVM (all the aforementioned big data systems like Kafka, Cassanda, HBase, etc.) A number of these systems have issues with the JVM, particularly GC performance on large heaps, and Rust would be very attractive here. Scala developers in particular are used to modern language idioms and so could relatively easily switch to Rust.
Oh, indeed -- and I might even agree on the non-platform arguments. And I'm not discounting those -- I'm just saying that the JVM (not Java/Scala/Groovy/whatever) argument is overstated :). IMO, as always.
Agreed about Rust as well, I think. Java programmers are already used to the verbosity (tongue firmly in cheek ;)).
EDIT: Actually, come to think of it... many of the criteria you mentioned would apply to JavaScript developers, wouldn't they?
http://www.oracle.com/technetwork/java/javaseproducts/missio...
https://www.jetbrains.com/idea/
https://wiki.openjdk.java.net/display/Graal/Main
http://www.oracle.com/technetwork/java/javase/downloads/java...
All in one package.
Plus since JVM is actually a specification, there are JVMs for almost all CPUs and OS that matter, from embedded platforms all the way to the data center.
VisualVM: Great, surely any VM should provide all of this through metrics or profiling? I can count the number of times VisualVM has told me anything I didn't already know or diagnose through other means on one hand.
Mission Control: Can't speak about it yet, haven't tried. (But I will!)
IDEA: Meh. It's a Java IDE. (Well, it does Scala too and some other stuff, but the thing is I don't actually miss that when programming in, say, Haskell. I do have some auto-compile stuff in my Emacs, but frankly I get along fine without it.)
Graal? Experimental at best. AFAIUI it's actually quite a lot slower than State of the Art JS engines. Don't get me wrong, it's still great to have, but it's not exactly a game-changer if I could just shell out to another JS engine and communicate via sockets. (Don't get me wrong, it's great research. It's also a new thing. I've been hearing this "JVM is great" meme for years and years.)
SceneBuilder? Seriously? If you've got OpenGL, why would you bother?
Yes, it's all in one package, but at what cost? You're having to program on a VM which lack even such basics as tail-call-optimization, value types, etc. (I realize some of these things may be coming, but I'm talking about now).
Standardization, I'll give you, but frankly, has the level of standardization actually been tested seriously? Can you take any old compiled-for-1.5 app and run it on Azul's JVM? Can you do the reverse? (I'm not actually sure what kind of guarantees the TCK does provide. In any case and in my experience, most "ordinary" admins are running the unverified OpenJDK on their servers.)
EDIT: Sorry, my spelling and grammar are atrocious when slightly drunk. I can only apologize.
EDIT#2: Sorry, if there were any ninja edits. I'll try to stop now.
> IDEA: Meh. It's a Java IDE. (Well, it does Scala too and some other stuff, but the thing is I don't actually miss that when programming in, say, Haskell. I do have some auto-compile stuff in my Emacs, but frankly I get along fine without it.)
I have been there, VI (not VIM) and Emacs.
None of them can give me the experience I had with Smalltalk and that modern IDEs try to built upon.
But to each its own.
> Graal? Experimental at best.
It is planned to eventually replace Hotspot and is better than it in escape analysis.
JRuby is now using it.
> SceneBuilder? Seriously? If you've got OpenGL, why would you bother?
What has a RAD tool for GUI have to do with OpenGL? Do you write all your widgets by hand?
> Standardization, I'll give you, but frankly, has the level of standardization actually been tested seriously?
Yes. We develop on Windows and deploy on Oracle and IBM JVMs.
I am aware of projects working on Aonix, JamaicaVM and many others.
I don't know what "ordinary" admins are. We do Fortune 500 consulting usually with multi-site development.
I'm happy for you to do F500 consulting, but I don't think you've actually come up with anything persuasive.
I can't say I've actually tried Smalltalk IDEs, but as I understand them they have a "repeatability" problem: could one and did you do distributed version control with your Smalltalk code? Other than copying images around? (I fully acknowledge that IDEs provide value for Java and Scala, but that's because those languages are fundamentally broken, from my perspective at least. This has nothing to do with the JVM, obviously.)
Re: Graal. Yes, perhaps in future. I'm talking about now. I can postulate any number of wonderful things that'll happen in the future... that won't acutally happen.
"Develop on Windows and deploy on X". If you have good enough QA you can get away with that, but I happened regularly on my (former) team that someone used "C:/whatever" as a path name in their configuration... which wouldn't work on Linux machines, obviously. Unfortunately not all of us have the luxury of working in an environment which has ample resources for QA and Integration Testing or what have you. I can fully agree that if you have enough resources, JVM is going to work just fine for you. I just don't think it's anywhere near optimal on the result/(effort+expense) scale.
What else is there? That compets in performance, langauge and tooling?
You still haven't stated exactly what it should compete with and what you imagine as the alternative to the JVM. Imagine I have absolutely no tooling, no language, no performance. Sell me on the JVM.
Now imagine you could have the world. What would your ideal VM look like?
"That compets in performance [...]": Well, C/C++ certainly competes in performance, Rust probably as well. Hell, even JS probably competes favorably on modern browsers. What I'm questioning here is really whether you can actually say "X is best for everything" or "X is best practice". (Where X was obviously the JVM.)
(I just mentioned Haskell as an aside, let's not make that the primary point of this exchange.)
It competes in performance if you are good at programming in it. It does not compete in securty, ease of use, buildsystems and complexity. Writing C/C++ code if you are not a export is basiclly just one big security hole, that is one of the major reasons.
> Rust probably as well
I love Rust but its no even anywere near what we are talking about
> Hell, even JS probably competes favorably on modern browsers.
Well 'on modern browsers' is very limiting. It cant compete on the server side, not even closly.
> "X is best for everything"
Nobody is claiming that, thats not what at issue. Its good at many things, and it specially good at server-side stuff.
I think JS is on the way to be a real competitor. The number of packages created for it grows faster than probably any other language. It is already fast on the server and more and more tools are being developed. If you count lack of static typing as show stopper then it can't compete. On the other hand Dart and TypeScript has types and might solve that part.
Only .NET, I would say.
Competition can only be good, seams to me they are pretty much a wash at the moment. For me the JVM is much better because thats the prefered Clojure platform.
- Always JITs, there is no interpretation. Some JVMs do this as well. On the reference implementation, you need to set interpretation threshold to zero.
- Tail Call Optimization. Eventually it will come to the JVM, after the bytecode verifier gets updated.
- Reiffied generics. JVM 9 or maybe 10, depending on project evolution.
- Value types. JVM 9 or maybe 10, depending on project evolution.
- Better interop story. Oracle is now working on a JNI replacement for Java 9, maybe 10.
- support for pointers in unsafe code
- support for stack allocation
- control over memory layouts
> Rich Hickey for example said he had much more problems with the CLR and ended up mostly dropping it (could be diffrent knowdays).
Quite up to date.
Control over memory layouts. That would be truly useful for getting reasonable performance. And dropping memory required by the reference hell. Java doesn't support arrays for anything but basic types. Just boolean, byte, char, double, float, int, long. And (object|array) reference.
Arrays in Java can have up to 2^31 entries. An array of 2^31 Objects takes 4 * 2^31 (8 GB), with compressed references. Plus of course 8 * 2^31 (8+16 GB) for java.lang.Objects that don't do anything. Smallest usable objects with at least one member would take 16 * 2^31 (8+32 GB) RAM. But this would prevent you from using compressed references (max 32GB heap), which would increase consumed memory to 16+32 GB. Like for example for 2 billion Integers. I realize you wouldn't store integer values this way, the point is integers are about the smallest useful objects imaginable. Java should have true object arrays (like C structs).
And how do you align objects to say a cache line? You don't! It's hard to even avoid false sharing in Java. Practically impossible, because you can even get cross-object false sharing due to object memory alignment!
Always JITs can be also a bad thing, why bother with code that is very rarely executed? If time it takes to compile + execute compiled version exceeds time it takes to interpret, why bother?
When it comes to the back end, I'll likely choose Scala 100% of the time. I really believe there is no substitute for a strongly typed ML-derivative for that type of environment. Additionally, there are a strong set of robust scala libraries (Typesafe mostly) that are completely production ready, and java interop is much more natural when you need it.
For the front end, I'm currently using Javascript, but I have toyed with the idea of both Scala.js and Clojurescript. This is where Clojure really makes me mad. I feel as though the dynamically typed Clojurescript should be a better fit as a compile-to-javascript language, but in every instance I've tried, Scala.js just works better. It compiles correctly, error messages make sense, much better documented, and there are fewer deviations from the core language. And the clojurescript libraries are mostly terrible in this regard...even the super-hyped ones like Om. My biggest pet peeve are the tutorials that force you to learn unrelated things (like Datomic, Emacs, Ring, etc) to be able to walk through them. I really feel like Clojurescript had a chance to shine here, but I find it far less inspiring than the relatively immature Scala.js. Oh well.
(*) Not that there aren't a thousand reasons to not choose Haskell. This is merely obligatory temptation dangling.
let P =
let S = { s ∈ Reasons | s : ¬ Choose Haskell } in (S, |S| > 1000)
in ¬(¬∃ P)Incidentally, I've finally started to sink my teeth into Scala with a larger project. So far my impression has been "Very nearly as good a language for functional programming as C#." It's got ML-ish syntax, but the Java-y OOisms are just way too front-and-center for comfort. Case classes and extractor patterns, I'm looking at you.
That's why I feel like something's missing from F# in this regard, in comparison to other statically typed functional languages.
Or to put it another way; Scala as an idea may be fine; you can mix imperative and functional together, by picking which data structure and approach for a given problem is 'right'. But in a team environment? Touching someone else's code in a particular paradigm means you have to be familiar with, and expect, that paradigm. You can come up with a team that agrees on what paradigm everything must be written in (and that can be anywhere on the continuum), but you can't mix and match and expect developers to not have to learn every paradigm in use.
All of those require being familiar with the functional paradigm, or a data type conversion. And vice versa.
Sure, for code that doesn't interact, or that does only through an interface not defined by the language (i.e., through the OS, through sockets, whatever), the issues in style don't exist, and Scala kinda makes sense as a "single language that tries to be all things to all people". But on a shared codebase, with a 'team style', per the OP, that allows people to use what they're most familiar with...mixing paradigms is going to cause problems, and will force each dev to learn each paradigm in use.
Please note I'm not saying there's no benefit to mixing paradigms, or that a language that includes multiple paradigms is bad, but rather that by allowing more paradigms the codebase you increase the breadth of knowledge necessary to work on that codebase, such that letting people stay in their comfortable, single paradigm bubble is impossible, at least if you want them directly interacting with the code written by someone else in a different paradigm.
And an ST monad makes no sense unless you're interacting with imperative code (obviously, hence your bringing it up), which means you already understand what imperative is, and are still having to convert your immutable structure to a mutable one, for the imperative code to operate on, and to recognize it will be modified in place; to use the ST monad the functional programmer already has to understand imperative programming.
It also seems like it would fail to encapsulate the side effects an imperative function may have, since that function is able to interact with state elsewhere, that isn't actually encapsulated. I.e., the ST monad makes sense when dropping into update-in-place for performance reasons, but when there are side effects that are not ensconced within the monad (remember, the function you're calling was written by an imperative programmer, in a language that allows side effects outside of a monad; not Haskell), the ST monad does nothing to make it functional, and oh crap, now your functional programmer has the same problems as an imperative programmer, in having to figure out WTF this function just changed in some other imperative chunk of code, in some other process or object or whatever.
So given imperative and functional paradigms, even using the ST monad, if the functional code is to call the imperative, and the imperative is to call the functional, both the functional and the imperative developers have to understand the other paradigm, -and- deal with the considerations of both. You've upped the complexity of your system considerably.
The ST monad also does exactly restrict the scale of effect of mutability which can occur. It's not a license for the imperative programmer to do whatever they want---it's a license for them to solve problems in the way they prefer within a particular ensconced region.
If you want, you can write a program where everything lives inside of a giant ST monad layer. You always do this if you replace ST with IO, in fact. Each bit which can be made purely functional will be embedded into this monad and, for sake of sanity and testing, those functional bits ought to grow and grow.
But it doesn't really demean the IO/ST segment. An imperative programmer can live happily there without much trouble at all.
I wasn't saying you couldn't set boundaries; you have them implicitly simply by dint of the fact that in Scala, mutable/immutable is based on the type, and the type system is enough to enforce that. But now you have a performance penalty being introduced based on who works on a given part of the code, and one that can easily compound.
"It's not a license for the imperative programmer to do whatever they want---it's a license for them to solve problems in the way they prefer within a particular ensconced region."
But for the imperative programmer to know what not to do he/she has to understand functional coding, and choose to eschew certain techniques that might be viewed as acceptable from a purely imperative perspective.
Please note, I didn't say you -can't- do it. Just that it's asking for trouble, and that either you have to understand both paradigms or you suffer a performance penalty (converting from mutable to immutable and vice versa). And that if you ever want to touch someone else's code you have to be familiar with the paradigm. Not proficient, but enough to realize what you have to do to be able to interact with it properly/safely, that that requires a broader depth of knowledge on the part of each team member than if you picked one paradigm, and that leaving them to know only their paradigm will not work. The imperative person has to figure out higher order functions, working with immutable data structures, etc, to make use of some of the stuff the functional guy has written, and the functional guy has to be comfortable with destructive updates, side effects, etc, to be able to make use of what the imperative guy has written. Yes, they both can interface with the other, but what they have to know is much broader, and every time they interact you risk bugs and performance penalties that you'd not have if you picked one paradigm alone.
It's a good solution in Haskell though.
"In well over a year of working in Scala teams there hasn’t been a single day where I felt that there was a shared mindset about how to develop a system or even approach a problem."
So this guy expects a programming language to create a shared mindset?
I've been developing software for close to 17 years in more languages than you can shake a stick at and none of them do this for you. It's always about picking the right tool for the job and/or team.
A shared mindset comes from other planning type activities like use cases, function/non-functional specifications, and general architecture type planning before you write code. Especially for larger projects.
I think software patterns tend to help drive synergy in a team environment. It's not perfect but my experience has shown me and the folks I work with at least, that adopting patterns helps to bring us together but those patterns are usually fleshed out during architecture planning activities.
I should note that I agree, and I'm a Python dev full time.
Speaking of third party libraries, you want to install those? Just go with distuils, no setuptools, no distribute, no distutils2, no distlib, no wait distutils again but a new one. I probably even missed a few.
It's still my go to language for a lot of things, but I can't help but agree that design goal hasn't aged very well.
Like you said, "software patterns tend to help drive synergy." I don't think your opinion and the authors are all that far apart.
Either you learn to live with code that works, regardless how it is written, in a world that doesn't care about stuff discussed in HN or better change job.
Trying to bring everyone involved, both onsite and distributed locations to follow best practices, is playing D. Quixote.
and ... switches to a lisp. You know, languages that execute code by rewriting the syntax tree of the running program ...
This ought to be good.
This is not meant to be disparaging against LISPs. I'm just alluding to the fact that while there are 5-10 ways to do the same thing in Scala, all looking very different, working subtly different, there's 5000 different ways to do the same thing in LISPs.
Scala, on the other hand, has wildly diverging sets of best practices. The feature set is huge, attempting to appeal to pure functional programmers in their academic ivory towers, and the grizzled Java developers who believe OOP is the one true way, and everyone in between.
It's easy to -say- that there are 5000 ways to do the same thing in a LISP, and only 5-10 ways to do the same thing in Scala (both statements being hyperbolic, presumably; there are infinitely many ways of doing each, after all), the meaningful variations you are likely to see in the wild are far, far lower in Clojure than Scala. And will likely be based on different understandings/expressions of the problem, rather than on different language subsets that the author prefers.
Not really. This best practice is for beginners and mid-level programmers. They either write macros for the wrong purpose or in problematic ways. Thus they should avoid macros. Still they need to use macros, since languages like Common Lisp provide a lot of functionality through macros.
To get past that stage a book like 'On Lisp' by Paul Graham helps.
Many, but not all, advanced programmers will use and write a lot of macros.
While 'On Lisp' is a cool book, I rarly find the really advanced macro stuff usful. Once in awhile, you want to have something like a really good pattern matching compiler, and then your macros come in handy. But day to day, you dont need them much.
https://github.com/Wukix/LambdaLite/blob/master/lambdalite.l...
This is a recent tiny in-core database.
The code is using LOTS of macros. It also implements a few non-trivial macros. Half of the code is implemented as macros.
Code like that is not unusual in Lisp.
There's a reason I qualified it with "generally" and "rarely".
That's not where we live.
Some of the importent keynotes from some of the early conjs are really importent, and they really went deep in the community, you can see it in almost every major library.
Another commenter has mentioned the notion of 'simple' but thats not really all that clear. I not enougth of a writer to tell you about it, but I can give you the resources:
- This classic got Clojure on the map for many people: Are We There Yet? - Rich Hickey www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey
- Simple Made Easy - Rich Hickey - http://www.infoq.com/presentations/Simple-Made-Easy-QCon-Lon...
- Simplicity Ain't Easy - Stuart Halloway - https://www.youtube.com/watch?v=cidchWg74Y4
- Ousterhout's Dichotomy Isn't (Part 2 of the Simplicity/Power/Focus Series) - Stuart Halloway - https://www.youtube.com/watch?v=cidchWg74Y4
- The Language of the System - Rich Hickey https://www.youtube.com/watch?v=ROor6_NGIWU
- Hammock Driven Development - Rich Hickey https://www.youtube.com/watch?v=f84n5oFoZBc
In addition to these, there are many talk that are about people trying to run with these ideas. Datomic is a example of taking these ideas to the max, it really standas as a nice example. You can find others in talks by David Nolan for example, but there are tons of applied talks.
- The Design of Datomic - Rich Hickey http://www.infoq.com/presentations/The-Design-of-Datomic
When working in large code bases with languages that don't have macros I find myself seeing people trying to accomplish the same task, but in subtly different ways. These are cases where a helper function usually would not reduce the LOC or complexity.
As with everything, macros can be bad, but I wouldn't be so quick to ban them outright.
Keep a language simple, that's how I like it at least. It's not about how many ways are there to write X, it's more about how many times can I write the same X and have it do different things? (aka ambiguity)
The answer in Lisp is exactly one: S-Exp, obviously there are exceptions like delayed evaluation, re-binding of symbols (not really recommended, at least in Clojure) and argument parsing in macros, but those are exactly that, exceptions. Take operator overloading in many programming languages or method overriding as a comparison, way too much ambiguity.
Because of how LISP is structured, it is extremely tempting to have dozens of them for any reasonably-sized program.
In practice, in all LISPs, this number of syntax forms is quite high. In clojure it's somewhat less, but still a lot more than in "average" programming languages.
S-expressions are a syntax for data. s-expressions are not the syntax of Lisp.
Lisp syntax is defined on top of s-expressions. In Common Lisp there are basically four forms of syntax:
* data
* function calls
* special forms, basic building blocks of the language. There are only a fixed number and actually very few of them.
* macro forms. Many provided by the language and user extensible. The user can write new macros or extend some existing macros.
The main differences between many languages and Lisp: the syntax is extensible via macros and most of the time it is relatively simple to see the implementation and the expansion of a macro.
Larger Lisp systems can contain hundreds of macros or even thousands of them...
http://blog.fogus.me/2011/03/27/the-long-lost-art-of-thought...
For (1): I disagree -- Scala's syntax is relatively easy to grasp. Perhaps the author meant constructs.
For (2): people keep saying this but why not just stick with the way you are comfortable with?
For (3): I don't see how this is a language issue; at the very least it shows that the author is unable to grasp a particular functional programming abstraction.
I think Go is an interesting language partly because they seem to keep it simple by choice.
To put it another way, under extreme circumstances, Mr. Martin Odesky can use even the $&^%$#$ PHP to write safe and efficient airplane flying control software.
The point is the language is hardly matter as much as your kids think. There are far more important things for good software. But language is not only a tool but also a toy and a passion, so I can appreciate the urge to talk about it. :-)
I think thats what he was going for.
> There are far more important things for good software.
Everybody knows this, but people still like to talk about programming languages.