F# Not just for finance
fsharp.tv
fsharp.tv
Specifically the article missed one of largest F# deployment, in production, in the world at this point. We use F# at Jet.com and it powers every part of our core business from our dynamic pricing algorithm to search and analytics.
Over 4 million customers already on jet and over 2200 cores on azure all running F# code.
GC on the other hand is very aggressive with all of the immutable data structures F# creates. GC in F# is very good, though, I think without good garbage collection you have a tough time in a functional world. Microsoft is especially interested in GC performance for .NET and they have explored memory dumps to improve GC so it performs well even under load and when used with F#.
However, both Async and IO are insufficient to represent disjunctions. To that end, another concurrency library that we use is Hopac. Hopac is an F# implementation of CML, with some differences. Hopac provides a notion of an alternative (called event in CML; think Haskel's Alternative typeclass if you relax the laws a bit) and synchronous channels (note that async channels are special cases of sync channels). Hopac's has experimental support for lawful MonadPlus as well (see transactional events in Haskell = IO + STM + CML).
Some things that we're heading towards next are generalizing STM to be a bit to be more like RCU (see relativistic Haskell). Additionally, we are experimenting with extending this to session types, but nothing in production yet.
F# also provides a MailboxProcessor, which is similar to an Erlang actor, however without explicit distribution support, so perhaps more of an "agent". We typically use this as a low-level concurrency primitive, rather than a full-blown programming model. Most of our services are compositions of various request/reply interactions, and the Async model above is a great fit for this. In fact, we've primitives centered around the notion of an arrow 'a -> Async<'b> (specialized to Async). These primitives provide support for fault tolerance, logging, tracing, etc.
All of this works well with the GC. Async does cause allocations of course, but this is a price we're more than willing to pay. We've shared some GC dumps with the designer of the .NET GC and she believes they are sensible for a functional language. Hopac took optimization to a greater extreme, reducing allocations where possible.
Another F# library of interest is MBrace. This takes the notion of Async and fits it with a scheduler that schedules across a cluster rather than an individual instance.
Hopefully this helps!
Async is similar to a Future (such as in Java), with the difference that Future produces a result once and caches it, whereas Async re-evaluates each time it is executed. It can be made to cache of course. A Future is more like a TPL Task, though IMO, Async provides a more predictable programming model.
Go has go-routines and channels. A go-routine is similar to Async. In essence, they are a notion of light-weigh thread. Note however that Hopac support for channels is far richer than that of Go.
You got it the other way around - synchronous is a special case of asynchronous, because any synchronous result or stream can be processed asynchronously, but for having guaranteed synchronous results you're adding restrictions. And going the other way, from async to sync is not possible without blocking threads, which is an error prone, platform specific hack. Take the possibility of blocking threads away and you'll notice the true nature of these models.
But yes, going from async to sync requires blocking, which is why CML/Hopac takes the approach of making sync the core primitive.
F# inherits lots of .Net's primitives which at their core are thread pool based task scheduling with the usual optimized data structures you see from this world. While this might sound primitive, F# takes it further by providing very good functional libraries which abstract this away allowing lightweight asynchronous computations to be passed around as values (the type being Async<'a>).
Alone, this is pretty nice but it wouldn't feel as natural without computation expressions which are similar to Haskell's do-notation but with some generalizations and flexibility added in. This allows asynchronous code to look and feel first class. If you're curious I'd check out https://fsharpforfunandprofit.com/posts/concurrency-async-an... or check similar links on your search engine of choice.
Now, the story isn't complete. If you're comparing things to Erlang, there are also things like MailboxProcessor<'a> and such which allow other common patterns to be easily implemented. The type checking here is a nice bonus over some other approaches to message passing languages. The big minus of course is that it's still a traditional runtime. If you really want Erlang's isolated processes, it's hard to do that well outside of a runtime like BEAM. I also think F# could use better fault-tolerance primitives but I'm actively working on this with heavy influence from my work with Erlang.
There is also the Async monad: https://msdn.microsoft.com/en-us/visualfsharpdocs/conceptual...
Ofcourse it is also possible to use standard .net stuff like the TPL https://msdn.microsoft.com/en-us/library/dd537609(v=vs.110)....
That sounds like a lot of cores for such a small data-set. Are you able to expand a little more on why you need so much computing resource?
The system is computing prices all of the time based on many factors. Plus we have built everything in house from our warehouse management system and supply chain tools to order management and everything else.
That amount of compute encompasses QA, Dev environments and any experimental and R&D work we are doing.
Plus jet.com is trying to compete with Amazon so that means the system has to be ready for many million more users than there are currently shopping with us.
And this is c++. It does handle a average of 200k requests per second though.
Don't underestimate true cpu intensive work.
We have a system that joins hospitals, primary and secondary care providers together. Each hospital might have it's own prices, as will their consultants, etc. Insurers have standard price lists, but some consultants and practices have direct deals with them, which can affect the price (and the creditor for the invoices), we also support rule-sets for modifying prices based on location, clinician, day of the week, availability, benefit maxima, credits, etc. We support the notion of inherited prices, where a standard template can be setup and then used as a basis for more bespoke plans etc. All of which can be manually overridden. If additional services are added then that can affect appointment lengths and costs. If the patient self-pays they may get different prices to what they would on their insurance.
Some of our customers do occupational health where they go out and get contracts with the big city firms to provide schemes for their employees. Each deal they get is different and negotiated to the nth degree, which can include various plans/tarrifs/credits, etc. Some services the patients can book themselves online, some can only be booked by their HR manager, some as follow-ups by clinicians, etc. Also rules around what can go on invoices, how billing items are grouped, who can see what (i.e. should an HR person see what services you've had etc.)
The additional complexity comes with matching that all up with efficient scheduling. We have some customers where they have appointments that involve up to 8 separate clinicians or resources; so they need to be scheduled in and priced without leaving massive gaps in the schedules. We also support call-centres that could see 1000s of practices and can query all of them. The end result looks very much like an airline booking model, but behind it all is a mass of complexity.
The combinatorial issues are large, and have definitely caused us some headaches over the years. So I fully understand how these systems can get very resource hungry. We're not in the same league as Expedia or Jet in terms of scale (although we hold more patient records than Jet has customers, if that means anything) - and as I mentioned it's relatively easy for us to silo groups of users, so I suspect some complexities just go away for us because we can filter down the potential results quite quickly, but still 2200 sounded like a lot - it was just a gut feeling. I don't understand how a shop selling a product can have difficulty calculating a price, but just as my customers have no idea of the complexity involved in calculating prices, I almost certainly don't understand the complexities that the team at Jet have to deal with.
The way we've dealt with it is to build massive data-sets of various static combinations, so if patient X on chargeband Y wants service Z then they'll be changed N. That does mean some large data updates when someone changes their pricing and rules; but it means doing less work live - although it can still be significant. This all runs on two 32 core servers.
I know healthcare insurance pricing has become the most insane invention of bureaucracy on the planet. That is why I can pay cash out of pocket and sometimes get an 85% discount. The providers don't have to chase their money from the insurance companies - private and government included.
C# for the core app, C# and F# for satellite services. Not sure on what you're asking re: platform. If you mean hosting? Then we roll our own servers in tier 4 data-centres. If you mean OS/framework: Windows Server/.NET
I keep meaning to look into Elm as it appears to be a more thoughtful approach to functional programming on the client.
I've played with Elm, but not made anything of note with it yet. The Elixir/Phoenix crowd are getting hopped up about it lately as a good fit for them.
That's 2k customers (not concurrents) per core, which is a terribly low rate. Even if they've only been running 1 month that's 1200 seconds (20 minutes!) of CPU time per customer, and it gets rapidly worse the longer you assume they've existed...
I'd really like to see some case studies going the other way: F# --> C# [or C++].
At the least, you could do C# --> C# [ie complete rewrite]. Any time you do a rewrite with more knowledge of what worked and what didn't, you'll have a better system.
So ultimately, the essay [like most] leaves the question: Is F# better? Or is it just that complete rewrites can provide substantial improvement?
If you were tasked to implement a project because first implementation failed, would you even read the first implementation or just base your work on the specification?
Should one read previous implementation? That's a difficult question. If you're worried about losing your perspective from seeing a single solution, then perhaps you're right that you shouldn't.
But it's possible that merely implementing it in a different [more functional] language would encourage sufficiently different patterns to allow you to keep perspective.
With all of this said, I am a huge fan of both C++ and Haskell. I had just had an internal realization that these stories might not be extracting the positive effect that a rewrite might have on a code base [even though, as you say, that is likely not relevant here].
I've since left and move to the Linux world, but have become more involved with using OCaml. Both are great languages (probably my favorites) and that's after investigating Haskell for a while. F#/OCaml's ability to easily move between functional/procedural/OO worlds makes it super flexible.
When I wrote the reference compiler for an intro to compiler class in OCaml, some of the teams who used Haskell asked me about my choice; I said that while most of the compiler was written in a nice FP style, it was nice to have imperative constructs in a few places to follow the techniques from a textbook without having to adapt them and making sure that I haven't changed the complexity.
Have you taken a look at F# on .NET Core? It's still in alpha, but aside from some of the FSharp.Core libraries you can write a lot of code with it.
also I'm just learning f# and I've started a small project to capture the paket and forge commands needed to get up and running: https://github.com/martindemello/fsharp-quickstart
The toolset is pretty boring/standard though: VS2015 + TeamCity + in-house Nuget + Gitlab
http://fsprojects.github.io/Paket/
It removes so much pain compared to raw usage of Nuget and is very easy to learn/adopt.
We have a very large number of projects and migrating them all to paket would be a hassle, so I'd be interested in what pain in particular it removes?
I'm no fan of nuget I can tell you, but I'd be interested to understand what you feel the big wins are
The satellite services communicate with each other through my LanguageExt.Process system which is a clustered actor-library, and so as long as the message formats stay the same they can be built and updated separately.
Each solution has a '<project name>.Dependencies' csproj included that contains all of the nu-get dependencies. It builds to ../bin, and then all of the projects in the solution add their dependencies from there. That means it's relatively easy to manage the dependencies for each solution.
There's a manual element of waiting for a project to build [on TeamCity] before updating the references in projects that depend on it, but on the whole it's not been too bad.
You can also integrate the restore part to msbuild so it works out of box from visual studio, msbuild, teamcity, whatever.
People prefer though to have separate restore packages step, the same way people prefer to use a better tool than raw msbuild to orchestrate build tasks (tool such as FAKE http://fsharp.github.io/FAKE/).
The proper way to do it with paket would be:
* get rid of the Dependencies project (it was a side effect of trying to cope with the pain of using raw nuget)
* add a paket.dependencies at the root of your repository listing all nugets used across the solutions
* next to each project, add a paket.references file with the names of the dependencies (only top level, you don't need to list transitive ones)
running the convert-from-nuget command should practically work out of the box if you don't have far out inconsistencies (which I believe you managed by reducing amount of packages.config files), but I'd recommend to do another round of sanitization similar to what I'm describing.
I think if you give it half a day of studying / using paket on a smaller repository, you'll figure out all the things you need to accomplish migration on a large scale repository, after which you'll be recouping benefits any time you need to adjust references in projects / add or update nuget packages.
ensure consistency of dependencies across repository (can't do that with so many packages.config files)
* easy to figure out outdated dependencies / update those with changes that are very easy to diff
* no need to battle with transitive dependencies
* several transitive dependencies resolution algorithm (max can be used to be as up-to-date as possible)
* ease support for multi platform targeting (it puts conditionals around the references, when you switch platform, it switches to correct assemblies without you having to do anything)
* will write/maintain binding redirects related to nuget packages for you (saves from many runtime errors)
* can generate include scripts for nuget packages if you do .csx or .fsx scripting
* many more things
the UI integration in VS is not as good but if you look at it, the UI is also part of the problem with Nuget (doesn't work on several solutions, doesn't really understand transitive dependencies, etc.).
You can migrate with single command line, and start improving from then on.
Tooling is what sucks. MonoDevelop F# support is very unreliable. Basic refactoring, like rename, don't work correctly every time and I had to literally do git reset few times, after renaming, as it screw up multiple files.
Also, editor often has visual glitches where letters get corrupted and I have to reopen the file to get it back to normal. Not something I expect from such a long time maintained application. However, it looks like Xamarin Studio 6.0 will be much better.
Alternative is Visual Studio Code with Ionide plugin, but I didn't use that beyond trying it once. Hopefully someone else can comment.
Edit: typos
Is it already released?
[1] https://developer.xamarin.com/releases/studio/xamarin.studio...
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb http://download.mono-project.com/repo/debian wheezy main" | sudo tee /etc/apt/sources.list.d/mono-xamarin.list
sudo apt-get install -y mono-complete fsharp monodevelophttp://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-ret...
OCaml and F# are quite close.
I would be cautious about drawing too many conclusions about languages he didn't end up picking; clearly he knows OCaml and Python well.
Should os.Getenv return an empty string or an error if the environment variable is not set? -- The answer to this depends on how unix-y you are, I suspect. We might disagree, but it's not wrong to say that you're going to do what bash does, which is, after all the fundamental place these are kept, and return an empty string. In particular, this is well documented in the API, regardless.
If I create an empty list, then marshal some uninspected text into it, and that marshal function fails but I don't check its error status, should my list be poisoned, or should it continue on as empty? -- The idiomatic answer to this is that you should check error status and remediate.
He would prefer that the program vomit when marshaling fails. Fine, but it was entirely his choice to code in this style, against go style guides. To then claim the compiler is 'unhelpful' later when he tries to read the list he declared is empty is blame shifting in my opinion.
I would guess he does not prefer in-line error checking and remediation as a pattern; that's totally fine, but it's annoying to read snark about it.
You say "The idiomatic answer to this is that you should check error status", yes, that's a Go design pattern, but nothing enforces that. In other languages, thanks to Option/Result types, this is enforced, hence his opinion.
Such kind of safety and security concerns should not be enforced by conventions.
I've had nothing but good experiences during my forays into functional langs. Here's to a more functional, immutable, easily-concurrent, easily-unit-tested future
While I like Erlang a lot, I'd say that Ericsson still has quite a bit of corporate control and interest. It's not always a bad thing to have companies involved in the success and advancement of technology.
However, dynamic typing turned me off. Whole my life I'm working with statically typed languages and heavilly rellying on compiler / type system to catch my mistakes / guide my design. Hard to give up on that.
Erlang started out in Ericsson, a corporation. Elixir and LFE (Lisp Flavored Erlang) started out opensource and are still opensource.
The term 'corporate self-interest' seems misplaced here, with the parenthetical remark turning it into the antonym of 'open-source'. The term proprietary, commercial or close-sourced seem more neutral and correct.
Erlang started inhouse at Ericsson, like F# did at MS Research, except it was for a company's immediate business needs or 'self-interest' to program their telecomm switches.
Elixir grew out of one person's frustration with Ruby's concurrency (Jose Valim), and a desire to have what Erlang offered him along with the BEAM VM and OTP. It has Ruby-like syntax, Jose is a popular Rubyist, and great tooling along with some other functional structures Jose added that he thought were missing in Erlang. [2]
Pony is an OO, actor-based, open source language, yet it has a lot of corporate pickup from fintech and others, and it seems to be getting ready to shove Erlang/Elixir/LFE aside on concurrency and speed. It has fully-concurrent garbage collection that doesn't use the "poison pill" message approach to kill all actors.
The creator of Pony, Sylvan Clebsch, has one foot in academia, and the other in business. He has worked on fintech, milsims, and games. [3]
[2] https://www.sitepoint.com/an-interview-with-elixir-creator-j...
[3] http://www.curry-on.org/2015/sessions/pony-making-it-easier-...
Unfortunately it is difficult to "prove" this definitively currently, until someone comes up with a "complexity polynomial metric over time" for code, except to talk to old programmers like myself (44) who have done this for a long while and gotten disillusioned due to all the time spent repairing OO complexity/tech debt bugs, and have become entranced by functional langs and the way they avoid inheritance, use immutable values and are super careful with state, all of which contributes to better long term code life
Fortunately I'm not the only old OO guy proclaiming this, I have John Carmack on my side:
http://www.gamasutra.com/view/news/169296/Indepth_Functional...
"My pragmatic summary: A large fraction of the flaws in software development are due to programmers not fully understanding all the possible states their code may execute in. In a multithreaded environment, the lack of understanding and the resulting problems are greatly amplified, almost to the point of panic if you are paying attention. Programming in a functional style makes the state presented to your code explicit, which makes it much easier to reason about, and, in a completely pure system, makes thread race conditions impossible."
"No matter what language you work in, programming in a functional style provides benefits. You should do it whenever it is convenient, and you should think hard about the decision when it isn't convenient."
The problem I see now is that the entire functional domain is so spread out - Clojure, F#, OCaml, Haskell, Idris, Erlang/Elixir/LFE, and even the APL/J/K/Q crowd and Java's attempt to go functional. I'm not complaining about the number of functional languages to choose from, which is good. It's whether there will be enough critical mass in any one of the functional languages given a new non-functional language popping up every week. I don't know.
Now that MS has opensourced so much, and Xamarin's stuff is free, F# is looking better to me (again).
I also find myself dropping to C. C was my third language (after 6502 assembly and CPM Basic). Haskell/Idris/Elm are my toys of the year. I never fully dove into Haskell, but the concise, readable and very mathematical syntax agrees with my sensibilities.
Old! I'm 52, but I didn't stay with programming for a living, so my age doesn't reflect my programming experience. I did start with a CPM PET in 1978/79, but then went on years later to do other things.
StackOverflow surveys, while interesting, are probably meaningless because they suffer from selection bias. Even so, I would guess that F# developers are very well paid, like other developers of FP languages, but it's probably not because they work with F#. The causality is likely reversed - good developers that tend to be well paid are also the kind of people naturally interested in expanding their skill set, hence interested in FP languages.
Nothing screams spam more than usage of a hot keyword like "functional programming" while leaving hints that you don't understand what you're talking about. I would expect an article that reads like a marketing brochure to at least make a short attempt at explaining what functional programming is. If you copy/paste testimonials from fsharp.org/testimonials, you could also copy/paste from Wikipedia. But then, their own course named "Functional Programming" doesn't seem to have anything to do with actual functional programming: https://fsharp.tv/courses/functional-programming/
Why do you think so?
Maybe you are unaware of that?
F# is a pretty good general purpose programming language and that is what the title and article tries to promote (albeit it doesn't give enough and diversified examples).
F# is also a reasonably easy programming language which puts emphasis first on "functional programming" (which you frame as hot keyword, I doubt the language was made to fill a hot keyword).
I don't know about the for pay contents of fsharp.tv, but I gather they might be introductory and try to bring understanding of "functional programming" aspects to an audience which isn't versed in it, it is sure good for them if "functional programming" is a hot keyword and people are looking for training material to pickup that language, but I wouldn't dismiss that as being spammy.
Chill out dude :)
In other words, when using a function the parameters of that function can be defined as other functions instead of needing to be precalculated variables or constants.
That's basically all that links functional programming languages, all other language design decisions can vary.
Even if that meant something in the past, it really isn't a useful definition any more, because pretty much all languages do this. That's why I offered my alternative description. Even if it too has flaws, I think it's a more meaningful description.
That's why I prefer to stick to the 'expression oriented' concept, it seems to be the thing that captures most, if not all, functional languages. But I guess just like there are plenty of people that wouldn't call Java OO (and would prefer the Alan Kay definition), there are plenty that will disagree with any definition of 'functional'
(preparing for the inevitable 'functional first' comment).
sremani != ZenoArrow (though if you read this sremani, thank you for attempting to clarify).
There are many 'multi-paradigm' languages, and C# is one of them. From the C# Wikipedia page...
https://en.wikipedia.org/wiki/C_Sharp_(programming_language)
"C# (pronounced as see sharp) is a multi-paradigm programming language encompassing strong typing, imperative, declarative, functional, generic, object-oriented (class-based), and component-oriented programming disciplines."
What makes a language 'functional' is first-class functions (i.e. functions that can be treated as values, which is what I was referring to in my earlier post).
As sremani said, F# is 'functional first', in the sense that the language is designed to make functional algorithms straightforward to express. You can write C# in a functional way too, but there's less syntactic sugar for this style of programming.
Hope that clears it up.
Apologies, my mistake.
> Hope that clears it up.
First paragraph on the Wikipedia page for Functional Programming [1]:
"In computer science, functional programming is a programming paradigm—a style of building the structure and elements of computer programs—that treats computation as the evaluation of mathematical functions and avoids changing-state and mutable data. It is a declarative programming paradigm, which means programming is done with expressions[1] or declarations[2] instead of statements."
I'll stick with my definition.
Hope that clears it up.
Ultimately, we have to live in a world where words are kind of ambiguous, but I think we should try to avoid useless definitions. The definition where "a functional programming language is one that can pass functions around" is extremely useless, because it applies to every modern programming language including C and C++. Whatever you want the term to mean, it shouldn't be that, because that distinguishes nothing.
It's not extremely useless when you consider most programming languages cover multiple PL paradigms.
Functional languages are a spectrum. On one end you have pure functional languages like Haskell where you're forced to use functional algorithms for all computation, but there are languages in the middle of that spectrum like C# which support a bunch of functional language constructs (take a look here for a selection: http://www.codeaddiction.net/articles/13/lambda-expressions-... ) even if they aren't purely functional.
My argument is, any definition of functional programming that ignores the mid part of the spectrum is useless, as it ignores that most languages do not strictly follow a single approach like Haskell but allow you to adapt to different styles (i.e. most programming languages are multi-paradigm). Even F# and Ocaml allow you to write in an imperative style if you so choose, and many people would class them as 'functional' languages.
No you're not. Haskell supports mutable, imperative, algorithms just fine.
https://hackage.haskell.org/package/base-4.9.0.0/docs/Data-I...
https://hackage.haskell.org/package/base-4.9.0.0/docs/Contro...
https://hackage.haskell.org/package/stm-2.4.4.1/docs/Control...
IORef's if I recall correctly offer the type of mutation similar to what is colloquially thought of as mutation.
These add to your argument a bit, but they're usually seen more of a last resort
Update: There is also a collection of good books, videos, etc. here: http://fsharpworks.com/materials.html
Also be sure to check out the F# Foundation site http://fsharp.org
Wouldn't the latter be implied by the former? Or am I missing something?
For example. Dart is the second highest paid language globally but doesn't even figure in the top paying US (maybe because too few US Dart devs responded).
It's sad.