“Mostly functional” programming does not work

Going to any extreme makes some things horribly difficult and going to another does the same for other things. So, optimally, multiple paradigms coexist in the single codebase, applied where they're most useful.

Functional programming with as many immutable bits as possible is definitely a good start. I generally do that for whatever problem I'm solving: I have a model for the data and then I write (if at all possible) pure functions to transform the inputs into meaningful outputs. But then you need know where to drop the ball and move over to some other paradigm that does something else right and merely drives the functional parts from the top level.

For example, such a data analysis library can be written with minimal state and using only pure functions but if -- and when -- you need some sort of an user interface so that the program can actually be used, an imperative/procedural approach is generally the most native approach because UIs are basically I/O. If you're adding a graphical user interface, you might use object oriented approach to build the UI tree which is probably the world's most idiomatic, canonical use for OO anyway. But even those are generally driven with an innately imperative event loop.

Also, note that the different approaches or paradigms aren't language specific either.

In the first stage, languages are tools that shape your thinking into accepting new programming paradigms but at some point you have a number of different ways of thinking in your head, and you can just forget about the languages they came from.

But in the second stage, you can just think directly in paradigms: you can consider different ways to build different parts of your program but you might actually use only one language to implement everything. You can write functional, imperative, object-oriented, and whatever code in C. Or you can use several languages with strengths in each paradigm, depending on what trade-offs produce the best engineering in each case.

There are ways to tame side effects without going monads, which don't really fix the complexity problem anyways (it just makes all effects explicit). See this paper for ideas on how to do that:

http://research.microsoft.com/pubs/211297/managedtime.pdf

We shouldn't treat state like an unwanted but necessary evil, we should embrace it and deal with it.

The fact that effects can be used to simulate other effects does not imply that programmers would typically try doing that. In fact, programming style is more often shaped by trivial inconveniences (often syntactic, or the availability of certain libraries) than most of us care to admit. A programming language that merely discourages people from writing spaghetti code should still cause less spaghetti code being written. Freak bugs will pop up from time to time, but not as often.

Shifting an existing language to a more functional style should be a net negative thing only if the resulting extra complexity and bloat gets excessive. Starting from an OOP/imperative position, it gets harder and harder to support an additional (marginal) piece of FP functionality, and after a while it's just better to start over and switch to pure FP, but current OOP languages flirting with FP are not particularly close to that point. They haven't started in earnest to introduce purity to an impure core language (which would be rather awkward, as the article points out). In this situation it is somewhat early to talk about mostly functional programming not working.

Scala might be closest to the point of excessive bloat, but I'm not really familiar with it, and I think that the FP support there is a good thing, especially in contrast to the Java code that it might displace.

(Disclaimer: I program chiefly in Haskell and I love it).

Back then there were legacy "structured" languages like C and Ada, and new exciting "object oriented" languages like Smalltalk and Eiffel. C++ was promoted as a "middle way" that let you "choose the best tool for the job". This made the pure OO languages look extremist. So, it was argued, if you had a problem best solved by structured programming you could do that, and if you were doing an application with objects in it then you could use those. It also meant that your old C programmers could pick up the tool and start using it immediately without having to relearn how to design a program.

"Aversion to Extremes" is a well-known cognitive bias, and these arguments play up to it, but of course it didn't work well in practice. OO features didn't dovetail neatly with the existing structured features, leading to an exponential explosion in the rules defining how the various features interacted. The mess was not helped by experienced structured programmers who felt they should use the new sexy OO features; the result was often a conventional structured design with some random virtual functions sprinkled around.

The book "Industrial Strength C++" is a case in point. It is basically a catalogue of C++ language features that interact in dangerous ways. http://www.amazon.com/Industrial-Strength-Recommendations-In...

Today we have the same story happening again. On one hand we have legacy OO languages like Java and C++, and on the other hand we have pure functional languages like Scheme and Haskell. So along come "hybrid functional" languages like Scala which basically make the same promise as C++: if your problem has lots of objects then you can carry on doing the same OO designs you know and love, but if you think that these magic first-class functions would be useful in some complicated algorithms then you can use those as well. And its going to fail for the same reasons that C++ failed: the OO and functional features don't interact well, so we are going to have lots of messy rules about them that cause subtle bugs, along with attempts by OO programmers to use chains of map and filter functions that work inefficiently because the compiler can't optimise them. And in ten years time there will be an "Industrial Strength Scala" book consisting of a long list of features that should be avoided if you want reliable software.

The infix application function (ma>>=\a->f(a)), commonly called bind, executes the computation ma to expose its effects, calling the resulting value a, and passes that to function f.

This is the kind of imprecise language that really made life extraordinarily difficult for me when I first learned about monads. I think this is an important point:

>>= does not execute ma!

If it did execute ma, that would violate the purity of the language.

Instead, >>= takes the computation ma and combines it with the function f to build a new, larger computation that is composed of smaller parts. The resulting computation (and ma) might never be executed (depending on the rest of the program).

I find it interesting that he's now advocating monads considering his earlier stance on static type systems[0]. Of course it may just be that he's changed his mind -- it happens.

(Yes, I consider monads as fundamentally requiring a static type system, at least if you're using monad transformers or similar advanced techniques. In practice you're not going to be able to get things right without compiler assistance when you have a stack of N monad transformers.)

I also find the bit on having a "pure" annotation vs. having explicit monads particularly insightful. The difference between a type system that only handles "pure/impure" vs. a type system that handles "pure/state-effect/writer-effect/network-effect/etc." is huge.

[0] http://lambda-the-ultimate.org/node/1311

Seriously? I don't even know C# and the "var q0" was enough to suggest the type of whatever Where returns is not an array of int (as opposed to the two functions above, with int and bool types), so why would I expect it to have filtered the array and returned it?

Ditto for the 2nd example: in this one it's even more clear that Select is returning IEnumerable<int>, not int[].

In C# the using statement causes the variable initialized at the entry of the block to be automatically disposed of when control flow reaches the end of the block. [...] surprising exception far away in time and space

There should be nothing "surprising" about that; maybe it is if you don't know C#. One of the first things that beginning C and C++ programmers learn rather quickly is never to return pointers to local variables or put those someplace where they'll need to be used after the function returns. How is this any different?

Imperative programs describe computations by repeatedly performing implicit effects on a shared global state. In a parallel/concurrent/distributed world, however, a single global state is an unacceptable bottleneck

Except that this "global state" is not really one thing, and it's not like all of its parts are modified by any one effect.

It appears that this "fundamentalist functional programming" the article is advocating is attempting to make programming more like math and distancing it farther from the real world by adding more abstraction, and if anything, I think abstraction is one thing that a lot of software these days needs far less of.

(Sorry if this is too rantlike, I have somewhat of a visceral reaction to these "the sky is falling!" style of articles...)

I absolutely agree that if you buy into Lazy programming, you have to buy into entirely Pure Functional as well.

However, many languages and frameworks have demonstrated a high degree of success mixing in functional paradigms (mostly centered around collections)

I would like to refer people to the concept of Collection Oriented Programming. In this paradigm, application specify most of there operations as mapping and reducing functions across different collections (trees, vectors, lists, etc). Not only does it promote safety, but it works in such high level constructs, it allows the compiler/interpreter to optimize the operation in many ways, such as optimizing out the lambda calls, and even parallelizing the operations.

To name a few Language+Libraries for which this is hugely successful: Ruby, Clojure, C++11 w/ std::algorithm, Scala, and Haskell, of course.

The solution, then, is not doing away with shared mutable state, as that's downright impossible, but with providing transactional semantics for that state. Once shared state has clear transactional semantics, the pure-functional nature of the programming language becomes secondary.

Let's imagine a programming language with perfect STM (i.e. an STM implementation that executes the most efficient code possible in every transaction). That language would have none of the problems described in the article, even if it were completely imperative. Hence, the problem is managing shared mutable state, and not existence of side-effects in general.

Pure functional programming could have been one solution to the problem, if only for the fact that it's not even a solution at all: Haskell programs still need a database. But the article assumes no other possible solutions, which is wrong. It focuses on one particular solution (which isn't even really a solution), rather than exploring many approaches. It is simply begging the question.

EDIT: I do agree that some partially-functional approaches are inherently dangerous, but I do not agree with the conclusion that the answer is going pure functional.

[1]: Except maybe for compilers, which is probably the most common complex software built in Haskell.

Here's a talk I was at a couple years ago where he makes fun of the ideas he's presenting in this article: http://www.youtube.com/watch?v=a-RAltgH8tw . Key quote: "obsession with monads is a medical condition".

I call BS. Pure OO and pure imperative has worked for half a century (a timespan in which functional languages have given us almost NO programs of importance, with the exception of Emacs, AutoCAD and a handful of others).

It's not like people have abandoned C/C++/Java/C#/Go/etc because they don't work anymore.

Plus, the need to get more out of multicore machines is quite exaggerated -- most programs can do just fine with just one core (if anything, they are unoptimized even for that). As for the others, programs like Premiere, Final Cut Pro X, Logic, Cubase, Maya, AAA games, etc, that is multimedia and number crunching stuff where performance is a premium, those are not done in functional languages (the particular examples are almost all C++).

As for high volume internet systems and services, those have found that Go/Scala/Clojure etc work well for them, to tap those cores.

So, yeah, "mostly functional", will do just fine.

> Recently, many are touting "nearly functional programming" and "limited side effects" as the perfect weapons against the new elephants in the room: concurrency and parallelism.

Who is this "many", and when did they say it was "perfect".

I think the premise is silly too. Even if you don't get the full benefit of functional programming without a hardcore functional language, you obviously get some. Limiting side effects is almost always a good thing.

If you "pragmatically" say, "Oh, but this is so hard, let's just let ourselves use a little bit of arbitrary-world-manipulation in our functions", you've basically returned back to the original world of programming with exponentially-complicated pieces again. As he points out in the article, even with a tiny crack in the wall, the compiler is back to being unable to assume purity. Programs must once again function as if an Int -> Int closure might read from the disk or hit the network. You're really back in the world of OO + old-school functional addons. I'm a bit more pragmatic and will agree that's a nice and useful paradigm, especially if you've learned discipline from time in the pure-functional world, but it is not the pure-functional world, and you will not reap the benefits.

(Which A: yes, they do exist and B: no, they aren't necessarily "mandatory" or the "only way to program". But still, see A. Personally I'd keep pure functional around for either applications that need high quality assurance without breaking the development budget, or programs of high complexity where the state space is big enough even before you start using horrifically unconstrained pieces to try to build your solution.)

I'd rephrase the title a bit... Mostly functional programming is not pure functional programming, and whereas I think "pure OO" didn't have a sweet spot where you insist everything is 100% OO, pure functional programming does. It isn't the only sweet spot. I think careful use of a very-not-pure language like Go, where one merely uses convention to avoid shared state (a very "pure" idea), can still be a sweet spot on its own. But there is another, where you go "purely pure", and for that one, you really do have to go purely pure, or you're not using it... and if you've never done it yourself, because you've only used impurely pure languages, you don't really have an opinion yourself because you've never tried it. Very nearly the only practical way to try it right now is Haskell.

In my experience, using imperative programming with elements of functional, is very productive, and I don't need to introduce monads everywhere to be more productive. Separating side effects, and making code as pure as possible, was a good practice in old style OO programming and will be so in the future, and introducing smart-sounding words for this which most of the readers don't understand (BTW, I do understand what monad is), is just a marketing trick.

APL element-wise operators are pure functions in an imperative language that has destructive assignment. However, it is also well suited to being used to implement GPGPU parallelism with local temporaries inside imperative procedures that appear to be pure functions from the outside.

Another form of parallelism arises from pipelined dataflow where tasks can simultaneously work on different parts of the whole linear computation so that those waiting for the results (further down the 'conveyor belt') receive more data from a potentially unlimited input stream 'just in time' whilst they consume more new data from their supplier or the source.

Taking things in the opposite direction a purely functional program can be seen as being a frozen moment in time that is subject to extrinsically defined constraints. Much like a spreadsheet these values can transition between epochs so they become ordinary mutable variables in a high-frequency event loop. This is the basis of exploratory "live" programming as an interpreter 'reacts' to dynamic changes in its source. It is also a facility provided by Mathematica where it permits the user to manipulate a graph of some complex function through recalculations based on new values of some sliders.

Every technique has its proper place and the latency incurred by Erlang mailboxes in the pursuit of fault tolerance and convenient hot-swapping of distributed modules is less of a performance issue when there would be a delay anyway given that the code is running over a network of computers. Erlang solves every significant problem of concurrency and it is highly reliable, unlike C# or Visual BASIC for which he is responsible. He really isn't in a position to criticise.

Both FP and OOP are extremes. Really, you can get by with Prototypes and have something type-oriented rather than class-based like Barbara Liskov's CLU. These can be given the capability of operating like Actors to simply take advantage of multicore / multiprocessor / multicomputer architectures. It helps your clarity of purpose if your language encapsulates persistent state in these Prototypal Actors without silly workarounds like C++'s friend function to make it go faster. A lot of your global state can live 'outside' of the program only to be seen from within as a set of global constants that change every 1/60th of a second when the runtime is reborn as if run from scratch with a slightly edited source. This may mean that apart from the output pipe of your dataflow that you can only put stuff IN to the BLACK BOX and must trust it to create its own views as to its current epochal value, just as a videogame renders a new frame of animation. All of the proponents of FP and OOP and for that matter Actors seek to prove that all programs can be written just using only their newly hyped paradigm. This is typical of ivory tower academia unsullied by the necessary pragmatism of the workplace. If you admit that there is something good about Actors then you don't have to learn Monads. If you admit that 'cloning' is a cleaner solution than the often abused (for the sake of convenience) 'implementation inheritance' and come to realize you can easily recreate classes / interfaces as abstract prototypes (i.e. they are just a pattern) you can jettison a whole lot of distracting OOP terminology and inscrutable UML diagrams as the work of self-promoting "architecture astronauts". Yet, if you embrace symbolic programming as seen in Mathematica you get to optimise your algorithms whilst not losing an insight into how their individual terms are transformed in your super-accessible declarative 'executable specification', at which point you realise that a symbol with an unknown value (i.e. a conventional mathematical variable) which can only become attached to a value once (per epoch) can be viewed as awaiting dataflow from a pipeline without any extra syntax obscuring your intent - you just call a variadic function and let the receiver await a finite list of arguments and then if there are more in the stream waiting beyond those it has already taken it awaits the same number of parameters again.

Really, the whole is greater than the sum of its parts - especially when those parts (paradigms) are dovetailed nicely.

I've been working on my own multiparadigm programming language for many years and Erik Meijer just seems too damn bleak.

http://code.google.com/p/kiama/wiki/Dataflow

That said, one of Scala most compelling (business) features--great Java interop--is also it's greatest liability. While I appreciate the great interop, Java's lack of state mutation controls in the language and JVM instruction set interfere greatly with the FP/OO impedance matching, generating misconceptions about the viability of hybrid FP/OO approaches.

From 1994 to 1996 I went from being a 80%/20% C++/Python developer to 90%/10% Java/C++ developer. My productivity went through the roof, but the lack of something akin to C++'s `const` references and `const` methods were a glaring mistake, one that remains to this day a shadow mandating contorted defensive programming techniques. Adding `final` almost made it worse, as it's nuanced and overloaded, and ultimately doesn't do what less-experienced developers think it does. Because of that one serious flaw, was I ready to go back to C++? Of course not, because I was able to render my ideas into working software at a faster pace--an extremely important factor--but I had move forward being ever aware of the language weaknesses and how to effectively ameliorate them.

Over the last year I've gone from developing in Java 90% of the time to Scala 75% of the time. In this transition I have seen a similar jump in my productivity (almost, but not quite as big as C++ to Java, and after a longer learning curve). However, have approached it with the same multi-dimensional awareness of one's paradigmatic assumptions and how they play with and against the language facilities.

For me, Scala was my gateway into the world of FP thinking, which radically changed the way I think about software problems. Scala has also had a profound impact in developing a deeper appreciation of the power of a more formal type system. Both of those paradigm-shifting features of the language have allowed me to be more creative, expressive and concise in my software writing, with bountiful rewards on multiple axes.

However, I've also stumbled along the way--becoming enamored of features I didn't fully understand, being too expressive when simplicity would suffice, being FP for FP sake, etc.--but I sure am glad I had those opportunities to stumble, and do so in a "fail-fast" manner. The process has been invaluable, and I'm a much better programmer today for it. I never entered the process assuming the FP/OO/CT academic visionaries or the Smalltalk/C++/Scala/Haskell/ML/OCaml language inventors offered me any "promises". They gave to the world constructs for others to think about and solve software problems, take it or leave it, to live and die in the ecosystem of ideas. I know it is my responsibility as a professional programmer to understand the pros and cons of those constructs and tools, weigh them against my goals, experience and intelligence, and go into a relationship with these tools knowing, I'm ultimately the one responsible for the final product, and need to know what I'm doing.

All this is to say, I don't think sweeping generalizations nor pointed nick-picks help in assisting people select the best language and paradigm for their problem at hand, understanding the strengths and weaknesses, and how to manage those trade-offs. It isn't, and doesn't have to be a "one-size-fits-all world" (as someone else here already referenced the great Stroustrup quote: "There are only two kinds of languages: the ones people complain about and the ones nobody uses."). And at the end of the day, the ultimate responsibility rests in the hands of the individual professional developer. In my developing awareness of the FP viewpoint I have most appreciated and benefited concretely from those pragmatic viewpoints in the middle. It is from the middle that one can more clearly see both perspectives, and from that develop a third, more holistic and encompassing viewpoint that harnesses the power of both.