After experiencing the joy of programming once again, I wonder why FP isn't widely adopted?
I understand in startupland that Django and Rails are simple to learn and prototype, but why aren't big companies and others using FP as a main language more often?
What are your thoughts?
Functional programming features are getting a foothold in mainstream languages. It's a drizzle, but the direction is clear.
Let's look at closures. Back in the 90s and early 2000s, if you were a mainstream developer using C, C++, Java or C# you were out of luck. But during the following decade all said languages (except C) added them, and nowadays closures are taken for granted in almost any new language being designed.
Functional list/stream processing APIs are the next feature being popularized, with such APIs being available in most mainstream languages nowadays (Go is a notable exception).
Other FP concepts like immutability, sum types, pattern matching and even type classes are finding their way into mainstream languages. Their adoption is far from universal but direction is clear. FP is becoming more widely adopted, just not necessarily FP languages.
Yup, real world engineering doesn't heed purity of concept. What you get is a hodgepodge of best practices and ideas that work. Today we see JS, Golang, Ruby, modern C, C++, all of these things have ideas and features directly drawn from functional programming and likewise, many functional programming languages use ideas from other paradigms.
We likely won't see Lisp or Smalltalk become dominant anytime soon but there is no denying that many of their features have made it into many of the main stream languages that have become mainstream for reasons far detached from their programming paradigms (um maybe not Java, for better or worse, Java sold OOD and OOD sold Java).
> all of these things have ideas and features directly drawn from functional programming
golang is the complete opposite of that, they disregarded useful research with unsubstantiated claims and it's just a basic procedural language with some concurrency, that's it.
The web frontend world has become very functional as well. The react stack is full of FP, for example. React was initially written in SML, then transcribed to JS and recently to OCaml/ReasonML.
The number one reason is that ultimately it feels too different, and Functional programmers tend to be terrible educators. So it all feels very esoteric. To learn a pure functional language is almost to start learning programming from scratch.
The secondary reason is the package ecosystem. Generally speaking these languages have a rough go at dependancy mgmt/build tooling (stack vs cabal and the lack of good, simple guides for their usage, and the same for bower(deprecated) psc-package and spago. The problem is that once the tools improve - the docs and guides are all outdated. Good reliable information takes more digging in these languages.
BUT it is achievable. These are perfectly serviceable languages that can provide robust, provable behaviors. And in the case of purescript, a strongly typed environment on the web.
The development processes are out there and there are folks writing full stack FP. (myself included). But as a whole, pure FP needs a lot more educational material in order to grow.
I think this is exactly it. When I decided to try Elixir, I figured it'd be just a bit more effort than trying out any other OO language. It hasn't been. It's been a matter of reconsidering just about every concept I had to solving a problem with code.
That said, it has been awesome and I love it!
[1]: https://en.m.wikipedia.org/wiki/Keirsey_Temperament_Sorter
What about developer productivity you might say? Yes a consideration - one of many. The last 20 years of language design has been focused on making people uninterested in programming programmers. That is what frameworks primarily accomplish. They waste memory and processor resources - are difficult to use in ways outside of their design. But any person can pick up rails for dummies and type gem something and presto a website. Believe it or not I think this is a good thing.
Putting on my Googler hat as well, there's a lot of work that's gone into our core libraries.
- Stubby, our RPC libraries, powers literally everything (Cloud, GSuite, Search, etc). I can trust that if I use it, I'm not going to be finding bugs in it - it's going to be some other big team that's got all sorts of fancy shmancy test that can and do catch strange, arcane bugs. The thought of relying on a brand spanking new implementation scares the shit out of me.
- We have a lot of powerful tooling (c.f. clang-mr, errorprone) that our core language teams use to constantly improve our codebase. That's years of tooling that a new language isn't going to have.
Taking the hat off, at the end of the day it's a tradeoff - either I spend my time continuing to work in a so-called boring language, or I learn a new one to use. If I choose the latter, I'm going to want it to be worth my time.
And frankly, sure, I don't get stuff like currying in Java, but Java 8's streams certainly do make life more bearable.
The cost of adding unexpected features or dealing with unexpected changes is directly proportional to the quality of the code-base, and in many cases a high quality code-base can make things profitable that otherwise wouldn't be (and vice versa). If software that has the ultimate goal of saving money actually costs money, I would say that it has failed, even if you ultimately produce fully functional software.
There was some kickstarter project a while back trying to make a game in golang and it failed because golang is not suited for that sort of stuff.
Functional programming lends itself to writing solid, verifiable code, and I trust these firms know what they’re doing given the amount of money involved.
Obviously you use the right tool for the job. For low-latency/HFT, it’s still probably going to be C/C++/ASM.
KDB is also fairly common at these sorts of companies and that was top of my mind when thinking of Fibonacci one liners. I've been through their youtube tutorials and various other bits of documentation and their are plenty of fibonacci like examples and precisely zero information on how to create a project, share code, deploy, etc. Smalltalk and other image based environments are much the same, they don't seem to have a lot of good answers for sharing code, or at least they don't show how. Another thing to keep in mind is what precisely they're used for at these companies, for some it's the core of the whole company for others it's just a fairly irelevent side project, I'd be willing to bet if you added up lines of code java would still come out on top by a mile. Our industry has a "you can prove anything with case studies" problem, see that one game built with lisp 20 years ago.
> Functional programming lends itself to writing solid, verifiable code,
I'm not sure that's true, it's a largely self selecting group that use these languages and I'm sure if you put them in front of an imperative compiler they'd be writing a lot of pure functions with little state. At worst imperative languages make it easier for state to creep in, but I consider that more of a feature in reality.
> and I trust these firms know what they’re doing given the amount of money involved.
I wouldn't be so sure, the industry has everything from old conservative companies still running COBOL mainframes to more cutting edge stuff, I know of one big progress user and even a large betting company building on nodejs. I personally worked on a 64-bit upgrade for a large company last year, finally they can run code on servers from this decade. And they're all held together by csv and tab delimited files flying in every direction, automated systems watching for FTP file drops, others scanning for email attachments and being run through a complex web of regexes. Not to mention the excel, if every copy of excel disapeared the industry would cease to function overnight. I've spent enough time in finance that keeping my money under a mattress seems somewhat prudent. There might be some good companies but there is no correlation between size/wealth and quality.
KDB as you say is common with financial companies. In fact I have only heard about it being used by financial companies. I have only used J, another array based programming system, and I found it brilliant for processing data. KDB has a niche in the financial world, and there is a reason for it. First Derivatives relies on KDB.
Obviously anyone who is doing anything interesting and profitable in this area is not going to be making it public, just like the world of betting.
There's another problem in the functional world, which is that the vast majority of publicly available talks you can find involve 90% wanking over the type system or how functional can beat C in some edge case (inevitably coded C-style) rather than focussing on the real world applications of what functional is suited to.
You're right that there are a lot of people in the functional world who have no idea about imperative, and could probably not tell the difference between a MOV and an LEA. I wager that these are not the same types of people who have actual high-paying jobs writing functional code.
The companies I mentioned have their own tooling, even their own compilers. They're applying functional programming to the niche it's suited to, namely providing a well-typed, functional interface to the more nitty-gritty systems, which are necessarily made up of Excel, COBOL and FTP file drops.
You're just not going to hear about this in the typical type-system wank talk/blog on Youtube/Medium.
This makes it tricky to correctly anticipate the outcome of a bunch of lines of code. One also needs to worry about the compiler, how it will interpret that, and this can be hard. Worse, the compiler's interpretation can change with a different version, or following a tiny source code modification that looked harmless. No one likes wild performance swings for no good reasons.
Add to that picture layers of abstraction that make debugging an executable more difficult, lack of choice and capabilities in said debugging tools (generally justified by "my code never has any bug!"), and it makes developing a headache, at least when things start to go awry. I haven't even started with 3rd party library supports, etc.
On top of that, one rarely codes in the void : a source code is going to be shared with a "community", so it's not a solo choice. One cannot impose a rarely used FP language to a team that know another language very well, and has never asked to change that.
Don't get me wrong, I love FP languages, and I can even see how some of their best concepts are being "borrowed" (ha!) by more declarative languages. That may be the best outcome. Maybe in the future, the differences will be less pronounced, the same way as RISC and CISC architecture have largely merged nowadays to inherit the best parts of both sides.
Every major language, like Java, C#, and JavaScript, is adding common FP features. Immutability by default is becoming more common.
Even React is FP-inspired.
It's happening, just not all at once.
JavaScript is the world's most popular language. Why? The web browser became the most popular application runtime in the world, and the only language supported by the browser natively for those applications is JavaScript.
Why is C the dominant language for systems programming? Because all operating systems used by the entire world are built on C.
If we were using Lisp machines everyone would be using Lisp. If browsers used a Haskell API instead of JavaScript API then we'd all be using Haskell.
C actually has merit, that's why.
1993 Donald Knuth interview: http://tex.loria.fr/historique/interviews/knuth-clb1993.html
"I think C has a lot of features that are very important. The way C handles pointers, for example, was a brilliant innovation; it solved a lot of problems that we had before in data structuring and made the programs look good afterwards. C isn't the perfect language, no language is, but I think it has a lot of virtues, and you can avoid the parts you don't like."
In functional languages this is inherently more difficult, especially when features such as lazy execution (Haskell) are introduced. A functional programmer has to reason about how the compiler will translate their unordered, pure statements into imperative machine instructions and stateful memory accesses in order to know how efficiently their program is going to run.
The above is of course an oversimplification - CPUs are complicated, and do all sorts of tricks to run sequential instructions out of order or in parallel when possible. Still, I’ve yet to see a functional language that makes reasoning about program efficiency as easy as in imperative languages.
I personally love Haskell, just not for most large-scale applications. As others have mentioned, because functional concepts can make code more succinct and more obviously correct in some cases, we’re seeing adoption of these concepts in imperative languages. I think this is good, pragmatic progress toward programming language nirvana, and I hope to see it continue.
These days, I don't think there is a main-stream language in use, that wouldn't have functions as values, anonymous function declarations, closure scoped variables and at least the standard battery of higher order function utilities, like map, fold/reduce, filter, e.t.c.
Closest comes Go language, with its "just use for-loop attitude".
But it is where all the innovation is (subjective opinion) and it's features are easier to port to other languages like JavaScript and PHP
Transducers for example solve so many problems with higher order functions it's insane
Persistent data structures solve immutability performance concerns
Datomic is a time travelling universal immutable database that solves so many problems verse SQL
Clojure has graph based end to end stacks thanks to Datomic Fulcro Pathom EQL
All of these things are amazing, not found in other FP languages and relatively new to programming as a whole
My rant on this is, is now I feel like I wasted a lot of time (years) learning Haskell F# ReasonML etc where FP appears to be conflated with a more feature filled type system but not much else when imo that's not where most of the value is
The problem is that computers fundamentally have both verbs and objects (data). And the real world has state. Managing state is a real problem when it is immutable and there is a lot of it. I'm guessing a functional programming network stack or image editor with undo would have a lot of performance-sapping copying going on.
So if you tend to have a lot of state, like UI, you probably want object-oriented programming. If you are munging through data once to get a result, maybe functional programming is better.
Or maybe your LISP enlightenment has been that you can run your data? I'm not FP expert, but it seems like that isn't the FP feature people people usually extol. I'm not even sure that FP languages besides LISP support it.
These data structures are called persistent data structures
Thanks for pointing out the use of persistent data structures, that is an interesting approach.
Not even close.
Haha. See what you did there.
An area in which object-orientation is particularly well suited, unlike functional programming. They are very much the reason for object-orientation's continuing success/ubiquity. Programming against "platforms" especially requires them.
* Writing extensible/overridable libraries and code bases that others can use (and understand).
Again an area in which object-orientation excels over functional programming. Consider why something like Node uses object-orientation to facilitate that?
FP is fairly widely used, as a programming approach. Using FP, especially impure FP of the type most Lisps are optimized around, doesn't require an FP language (pretty much any structured or OO language with first class functions supports it, and functional idioms have been driving features in OO languages for many years.
> why aren't big companies and others using FP as a main language more often?
FP is not a language.
The "mathematician mindset" needed to program in Haskell is alien to me. I like the idea of immutability etc. but there's just too much friction to get into it.
At least with Erlang it still feels like programming, Haskell just feels "wrong".
[1] https://homepages.inf.ed.ac.uk/wadler/papers/how-and-why/how...
Functional programming isn't taking over because it's not the best fit for all tasks. Each programming task needs the best tool for the job. Multiparadigm is the way to go.
First, the purpose and action of the code in question is immediately apparent to the (typically clueless, in my own case) reader. A for (or, heaven forbid, while) loop can do anything at all, but a map is quite explicitly transforming values of a collection (and/or turning them into a collection of side effects), whereas a filter's only job is to, well, filter, and a fold/reduce only needs to be conjured up when the programmer actually needs the full computing power of a generalized aggregation--quite the rare sight when good FP languages and libraries helpfully provide restricted (more descriptive and weaker!) versions such as sum and concatenate.
Second, using weaker functions means that the programmer (again, in my case, just an utter rube who needs all the help he can get) is saved from a breathtaking assortment of errors. Even in the chaotic anarchy of a dynamically typed language, a map will ensure you don't make off-by-one errors in your iteration, a pipeline of transformations will let you verify you did your processing in the correct order, and fold will at least give you the friendly encouragement to think hard about your initialization condition, and--more to the point--you won't even have the ability to carry forward state while iterating through a list unless you specifically request it.
The general point here, is that functional programming is trying to stop you from thinking about "looping" as a single abstract process, and instead have you explicitly reify the control flow of your program as higher-order function calls: this allows you to be explicit about what is supposed to be happening (to the benefit of both the reader and the writer, listed here in order of importance). Even better, in the presence of a Real Type System, this allows the compiler to actually tell you if your execution logic has the same general shape as your mental model, which is quite frankly a magical feeling once you see it actually happen when solving a difficult problem. That all being said, these benefits aren't free (as can be confirmed by anyone who has read a paper/tutorial on generalized recursion schemes...), so if you're able to write clear, maintainable, descriptive code while avoiding uninitialized states and incorrect boundary conditions and accidentally skipped elements and missing base cases and all the other horrible violence that I, personally and repeatedly, have inflicted on the poor algorithms I've clumsily attempted to implement, shine on you crazy diamond you.
Some clojure teams are moving to elixir because elixir is functional, performant, but imperative-y enough to seduce imperative programmers.
Edit: I’d nominate as a good definition of pure functional programming the absence of undeclared data flow (i.e. side effects) by your code.
Yes you can perform an SSA-versioned-variables or linear-types-changing-the-world transformation. That's just sophism; we can conclude that C is pure functional via the same line of argument.
In response to your edit: Rust does not encode side effects in its type system and does not require system calls to achieve side effects. If that's your point then we're in agreement.
We can imagine a language in which these don't exist, but it's not Rust and would not enjoy the ergonomics of Rust. That's why Rust is not pure functional, or even "almost" pure functional.
> but it's not Rust
That's why I stipulated Rust without UnsafeCell.
Businesses also like to isolate and manage their state in certain areas. OOP more accurately reflects this.
Although functional programming is becoming popular when scaled-out, take for example lambda functions or docker containers, which generally don't use state and nicely suit the functional programming paradigm.