The road to OCaml 5.0
discuss.ocaml.org
discuss.ocaml.org
It is probably one of the most underrated programming languages. The perfect marriage between state of the art functional programming and pragmatism. A great static and strong type system. Solid performance and an insanely fast compiler. Also compiles to JS if you need that.
Multicore support will make it quite perfect. Only thing that is holding it back more than that and the reason I have not done many projects with it, is it weirdly fragmented ecosystem.
Having to decide which standard library to use is a pain but you can cope with that. Tooling is getting there but stuff like automatic code formatting solutions are still pretty immature (and have really weird defaults).
Frontend there is that ReasonML/Reason/ReScript thing that Facebook it trying to do. It offers an alternative syntax but nearly nobody uses it because they changed the name and I think also the syntax three times already. So it is all a mess.
Don't let that stop you though. There are some pretty solid mature libraries in OCaml and if need be interop story with C and other languages is solid.
It honestly seems like a great lang, and I hope I get to try it out for a project sometime soon.
Maybe you are referring to the Async/Lwt dichotomy? Hopefully with multicore (and basic support for "effects" that are going to be merged in OCaml 5.0) this will become less of an issue going forward. Since the runtime is becoming considerably more capable, I expect there to be less "real" fragmentation going forward as the libraries begin to use more of the primitives provided by the runtime rather than building their own from scratch.
But then again, fragmentation is a way of life in other ecosystems too. Haskell has an ever increasing number of effect systems and preludes, Rust has many async runtimes also (async-std, tokio etc.). Fragmentation can often mean a time of competition and vitality as different approaches duke it out.
Regarding syntax -- I feel too much time has been spent in the OCaml ecosystem on surface syntax. OCaml syntax has its flaws but syntax is really a small aspect of the overall art of programming. The OCaml format is here to stay -- even if it is a bit wonky. Once you commit to it you can begin to worry about more substantial things. The ReasonML community brought in the new syntax, but with the departure of Rescript (Bucklescript) from the OCaml community I expect the usage of the new javascript-y syntax to decrease.
(If I may be heretical, I actually prefer the traditional OCaml syntax! ReasonML tries to be like JavaScript with the braces and so forth. I prefer the Haskell/OCaml syntax to the JavaScript/Rust/C/Scala brace syntax. Interestingly, Scala 3 allows a braceless style in an effort to match Python perhaps. Fashion changes. Algorithms and programming patterns endure. We shouldn't worry about the syntax so much -- as long as it is not APL ;-) ! )
I wasn't aware anything like that is happening, is it? Rescript departed from Reason, that's all, right? They want to focus solely on js target, because... that's what rescript is. New stuff they're doing looks really good.
ReasonML aims to supports all of the OCaml syntax while Rescript's aim is to preserve only that syntax that is beneficial for a JS target.
ReasonML syntax can be used to do backend OCaml programming as well as front end programming. Rescript is focussed only on the (browser) frontend and drops some features from ReasonML. Its all quite complex and confusing but the TL;DR according to me is that ReasonML syntax is not going to be as important in the OCaml community as before...
ReScript is very active. ReasonML is dead for quite a while now. I'm pretty sure ReScript will drop ReasonML syntax soon?
This is really about opinion and individual assessment. It depends on how much emphasis one assigns to surface syntax vs deep language features.
Yes, the surface syntax of Rescript has been cleaned up a bit. There are some new language features but the language is still "close" to OCaml/ReasonML as of end 2021 according to me. If you're able to understand OCaml/ReasonML today you won't have much problem reading Rescript.
But, yes, it is diverging and it could look very different in the coming years.
Things that don't make sense in javascript runtime context should not be supported.
Backward compatiblity on OCaml side is also something they don't need to care about as its ephemeral - it must typecheck and spit out valid javascript code, that's all.
OCaml has an academic flavor -- maybe it's not as academic as Haskell but it moves in similar ways. There is a desire to be correct and have a theoretical framework instead of amassing a ton of language features. OCaml is the foundation for Coq and other interesting compilers, type checkers and theorem provers. Over the years, the language has grown more mainstream and you can build a decent web backend on it today, for instance.
So fine, maybe multicore won't change adoption of the language in significant ways. But I foresee that the introduction of multicore will allow some amazing software to be written in OCaml in the future. Software that is truly groundbreaking and innovative. Take the example of Coq itself -- it is an important foundational software today in Computer Science. Multicore will allow Coq to potentially speed itself up and that will bring more real world applications in the ambit of Coq.
I wonder if that's precisely why people use it. I've been thinking about it, and I think people using OCaml value independence a lot. That's something that doesn't help building a community, since communities often thrive on consensus. As an example of that in the linked thread: Yaron Minsky's second comment about Flambda 2, which I'll copy here:
> And, I should add: Jane Street’s intent to upstream our work is not the same as upstream’s intent to accept it. None of what I’ve said is an announcement on behalf of the core OCaml team, nor am I in any position to make such an announcement!
This comment, to me, speaks volumes in terms of respect for the independence of the OCaml team. And independence seems to be something Jane Street values a lot too. They have lots of libraries that they freely share with other people. If you want to use, in a way, their "flavor of OCaml", you're free to do so. And if you don't want to, you're free to do something else.
You can see the same thing with JSOO, ReasonML/Reason/ReScript and now Melange. You're free to pick what you want. Same thing with the multicore. You want to use it? Great! You don't want to? They are working hard to make sure your code will still work and won't suffer too much performance regressions.
It may be a bit weird if you're used to other communities, I know I took a long time to understand why things are this way, and I may still be completely wrong. But I think the angle of valuing independence explains a lot, and is also a good way to know if it's a language and ecosystem for you or not.
Another thing that may not help: the book "Le langage Caml" is a great introduction to the language and programming, but sadly it's not translated.
"Nobody" uses ReasonML / Reason.
Plenty of people use ReScript.
I wasn't a big fan of the ReScript split, but I think by now it's unfair to speak of these as if they're one community with a confusing story. I think it's very fair to say it's now two separate communities: ReScript and OCaml. It was very confusing for a while, but by now it actually is much easier to understand than before the ReScript split:
ReScript is really its own language now. The compiler for that language just happens to still understand OCaml syntax, for now. The ReScript language and community is focused on the JS ecosystem, with readable JS output.
OCaml has js_of_ocaml. JSOO compiles OCaml to JS, so it's focused on the OCaml ecosystem. The JS output is not readable but you can build "any" OCaml program.
Really that's the main story -- not so hard to grasp?
There is also melange, but that's a relatively new effort in the OCaml community (attracting OCaml-y refugees from ReScript) whose status I haven't formed a view on yet. The idea is to compile OCaml programs to readable JS. Reason used to do that, but Reason now only has a tiny community (and I believe it now uses JSOO?). ReScript still does that, but using it for that purpose is no longer supported.
I think another important point is that the compiler is a fork of the OCaml compiler. That means that to contribute/maintain the compiler, you need to know OCaml. This is probably going to stay this way for a very long time, since the speed of the compiler is important.
> Really that's the main story -- not so hard to grasp?
What's not helping is that the people around Reason never really said "it's dead, move on to OCaml or Rescript". The pages for things like Reason Native, Esy, ReasonML are still up.
Important to the maintainers, absolutely -- but not directly important to users of course: it's firmly the goal that you don't need to know any OCaml to use ReScript (and I think it's mostly true by now).
> What's not helping is that the people around Reason never really said "it's dead, move on to OCaml or Rescript". The pages for things like Reason Native, Esy, ReasonML are still up.
I assume they'd say they're not dead. I think Jord Walke, for example, didn't want to break his commitment to existing users using it for native projects, so put in effort to help people switch to a JSOO-based Reason (I haven't followed that story, so not sure how that turned out). I think that's good behaviour.
They certainly have a far smaller community than before though, because most people went to either ReScript or OCaml, or kept a foot in both communities for different uses (and no doubt some left entirely).
That's true, and according to the maintainers, with the recent improvements it should be easy to maintain. And using tools written in other languages is not a first for JavaScript users (ESBuild comes to mind).
But it to me seemed packaged like many languages in the days of yore, when a language shipped simply as a compiler, and nothing more. The way of the world today to me seems to be a compiler, together with a complete standard library and consistent packaging system.
My experience with OCaml was thwarted repeatedly by a byzantine exploration process of packages depending on other packages, which required other packaging systems. Once I reached that point where it felt like I was spending more time figuring out the complex ecosystem, rather than writing code, I rapidly lost interest.
And perhaps such a point comes in exploring any new language. But it came much too early for me in OCaml. I had so much more I wanted to learn, but couldn't. I am hopeful for the new release. Thank you for your efforts, OCaml team.
Landing page: https://web.archive.org/web/20191002202720/https://ocaml.org...
From which you can click through to the Install page: https://web.archive.org/web/20190819032815/https://ocaml.org...
Over there the second and third lines are:
> The OCaml compiler and libraries can be installed in several ways:
> - With OPAM, the OCaml package manager (recommended).
That points you to the opam install instructions, which looks pretty similar to what it does now.
Look, I agree with you that OCaml installation and tooling are not the easiest to get into. But it wasn't like what the GP was making it out to be.
> Look, I agree with you that OCaml installation and tooling are not the easiest to get into. But it wasn't like what the GP was making it out to be.
I disagree with your interpretation of the initial post. Notice the "But it to me seemed" and "My experience with OCaml", and also "The way of the world today to me seems to be". These things are important. They inform us that that person is not exposing a truth, but sharing an experience, and personal preferences. They are also, I think, a way of saying "I don't know if the installation process was too hard, or if it was me that didn't understood something, but what I know is that in the end it didn't work out.".
So at this point, you have multiple options. You can empathize, you can work hard on trying to solve the issues on your end, assuming they are on your end, which might not be the case, you can ignore the post. What, I think, you shouldn't do is to deny this experience and imply that that person is "unreasonable".
I didn't have the experience you described (not yet anyway!).
Just to call out expectation-setting here in the comments: yes, the MVP of multicore will ship in OCaml 5.0, but OCaml 5.0 will ship no sooner than March 2022 (and very likely some point later, based on how challenging it appears to be to integrate the large-scale changes for multicore).
"Never have I took so long, to write so little code, that does so much"
OCaml can be a big learning curve, but I urge you to push through. The syntax might not be everyone's cup of team, but you get used to it quickly.
There's too little of it! OCaml seems to take a "you don't need syntax except when you need syntax" approach, which I found very destabilizing. One of the major online OCaml tutorials said something like "If it doesn't work the way you expect, try adding parentheses", and I thought "Oh hell no. In a Lisp I know exactly how many parentheses I need: all of them". I prefer not having to think about it, and letting the parentheses become invisible to me.
But otherwise I have a deep and irrational fondness for the language, and still wish I'd been able to make it stick.
That sounds like what I did with C++ with * and & when I didn't understood them. Do you think it's a lack of exprience/comprehension on your part, or that some parts of the syntax are fundamentally flawed?
But so much of the syntax, particularly around function calls, is simply a long row of whitespace-separated tokens, and I feel like my brain has to do extra work to parse what's what, and figure out associativity, and constantly remember what the ~ and the ? are doing. This[1] section of the tutorial makes perfect sense when you read it, but that doesn't mean you can easily scan a long function call with a lot of arguments, and instantly see what's happening.
The block-level syntax is great. But it got to the point where if I didn't write/read OCaml for a few days in a row, I forgot how it worked. And that's simply calling a function, nothing esoteric.
[1]: https://ocaml.org/learn/tutorials/labels.html#When-and-when-...
> The syntax for labels and optional arguments is confusing, and you may often wonder when to use ~foo, when to use ?foo and when to use plain foo. It's something of a black art which takes practice to get right.
So I don't think it's just you.
How is the python-ocaml interop story? To be clear, any language that does not have first-class interop with python is basically dead in the water (at least for our case).
I challenge you: A lack of understanding about the data lifetimes in a program means lack of understanding about the data.
Not saying you can't have a lot of short-lived data items that you don't want to manage one-by-one. I'm saying that for the vast majority of data items, one should be able to give a reasonably well defined lifetime upper bound. So a good solution is to make a few boxes that group items by lifetime. And from time to time, throw the outdated boxes away.
And of the few items that don't have such an upper bound at creation time, many can be created in a special box that allows migrating boxes later when required.
But this argument can extend forever.
Is your program precisely dependently typed? If not is that a lack of understanding about the nature of the data as well and should you challenge yourself to fix that?
You have to trade-off how much you specify things with how valuable it is to get the result more quickly.
You're just not going to get this buy-in from people who want to use a tool to get their work done.
Imagine you are trying to establish whether there's a relationship between timeseries X and timeseries Y. You just want a tool that allows you to quickly calculate some summary statistics of these timeseries, clean them, convince yourself that they behave according to your expectations and then run some form of regression.
Nowhere in this process do you care about lifetimes. It's literally irrelevant. In fact, as long as all your work fits into memory, you don't even care about memory management. Your objective is to answer the primary question, everything else is a costly distraction.
The 1% of ideas that ends up being worthwhile is what gets productionized and needs to be robust. But obviously rewriting everything from language A to radically different language B adds it's own headaches.
Typing does help, agreed.
With type annotations, you can move away from "inheritance OO" to logicless "data classes" and functions that operate on them.
Java does absolutely not have "good to great FP" support. It's an imperative and OO programming language that recently got lambdas, no more, no less.
- pipe operator (or custom operators in general)
- do-notation
- tail call optimisation
- currying / partial application
- (better) type inference
- expression orientated
- less syntactic noise around function calls (fewer commas and parenthesis)
- type-classes or even runtime generic information to work around that
Some of the thing I’ve listed can never be added without dramatically changing what Java is. At that point it would be a new language.
To get back to the subject of "good to great FP support", maybe we disagree fundamentally. Here are a few examples:
I would say that Rust is a good example of "great FP support". It's not a functional programming language, but it supports a lot of the patterns that make statically-typed functional programming languages effective. However, its DNA is still mostly "imperative programming language".
For "good FP support", I'd say TypeScript and C# are nice examples. C# has LINQ, pattern matching, records, optionals in a way. TypeScript has the usual JS stuff, plus the type system is nice. Both are lacking in some places. For example, in TypeScript/JavaScript, the usual map/reduce/filter are only functions on Array, and don't work with iterators.
To take the opposite view: what would be a functional programming language with good or great support for another paradigm? OCaml could be an example of good/great support for imperative programming. Scala has great functional programming and great object-oriented programming.
def calc_xxx(df:pandas.DataFrame) -> pd.DataFrame
Sometimes that is great. Other times, that will be very hard and error-prone.
A tool that makes something much harder without any meaningful gain should be avoided. Rust provides the tools to not have to fight the system and they should be used in these situations.
Once you have to think of types and lifetimes, a lot of the productivity goes down the drain.
99% of the stuff you do in research ends up being consigned to the cutting floor because it doesn't work. The 1% that ends up being useful is the only part worth productionizing.
In my case, I'd say that yes, I have to track the types myself, but the tradeoff is at least sometimes worth the extra mental overhead in my opinion. I'd say that the same can be true for lifetimes as well.
In your case, you will definitely be tracking lifetimes on some level and to say otherwise is going to be false (even GC'd languages must track lifetimes to ensure garbage is eliminated). The question is about the mental tradeoff vs the time taken. I'd guess that you are correct in your assessment. My only real point is that there is a cost that should be considered.
Research is fundamentally different from the usual programming exercise. Research is like prospecting. You want to try as many different locations as possible. You don't want to build nuclear shelter grade construction at every potential site because 99 out of a 100 is a dud. You'd never find anything.
You want a tool that allows you to get quick results to confirm if a site has potential and then you want to be able to scale your tools for proper mining.
That said, complicated Python systems can be improved a lot by adding type annotations. That's more of a solution for web servers and other "easily type-able" applications. Typing support for scientific computing isn't quite there yet. So it depends on what kinds of systems are the complicated ones.
Really a remarkable feat of engineering. Here's its author giving a talk: https://www.youtube.com/watch?v=vQPW16_jixs
Core.logic and the like opened a huge door and similar ideas have exhausted my free time for several years now.
CL could also be great language-wise (https://digikar99.github.io/py4cl2/, https://github.com/snmsts/burgled-batteries3) but I don't know how good the interop is in reality since I haven't tried it.
For that reason, Julia is being closely watched, but so far we are not thinking of pulling the trigger.
- Instagram was at one point a startup. It's common for startups to write in a scripting language to optimize for speed of adding features. Then if you grow or get acquired, the scripting language often eventually gets replaced by a language more optimized for maintenance cost, safety, and/or speed.
- Data scientists often only really know scripting languages, and at any rate scripting languages are useful for prototyping algorithms that need to change daily. Hence a lot of hedge funds use Python. For code that is stable and really matters for performance purposes, it's common for funds to use C/C++ or even FPGAs.
A skilled carpenter can undoubtedly use a hammer instead of a screwdriver. This doesn't mean that I should insist they use a hammer when a screwdriver would do a better job.
It's a dynamic, garbage collected language. It's easy to pick up and get going with. As a functional programming language there isn't a lot to learn in the way of language constructs, and you don't even have to do the 'wrestling with the type system' thing that you have to do in compiled functional languages like OCaml or Haskell (like you do in Rust).
Its processing 'horsepower' is probably comparable to Python, but it's much better for building low latency things if you want to run something in a bit more of a production use case. This is also improving due to the recent addition of a JIT.
The addition of NX is making Elixir an increasingly interesting place to do ML - write Elixir, have it run on GPU etc. See https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-...
Python integration is probably best done using the Erlang 'port' system - running Python as a managed process and communicating with it using messages over stdin/stdout. I use it for C interop and it works well (and fits well with the Elixir/Erlang process model). It's not difficult to roll your own in Python e.g. https://github.com/fujimisakari/erlang-port-with-python/blob... or look at something like http://erlport.org/
The main use case for a language other than python is a more robust codebase but also performance. We need to be able to efficiently ship lots of large arrays between the languages and the Rust-Python interop supports zero copy arrays for example.
But I agree somebody has to put the work.
I've made a good career with Elixir but I still don't think it's a good fit for a hedge fund. IMO invest in Rust.
https://signalsandthreads.com/python-ocaml-and-machine-learn...
No need to always go through the slowest path.
> Having said that, Ocaml compiler is one of the greatest miracles in PL when it comes to speed vs complexity of the language. Scala/Haskell/TS are not even close.
Someone will probably come correct me but what I've heard is that the compilation speed partially comes from the Pascal/Modula-3 influence, since Niklaus Wirth took compilation time into account when designing programming languages. From what I understand, OCaml doesn't allow circular dependencies outside of a single file, and that helps. Go doesn't allow them too, and is also known for its compilation speed.
You mention that you write a lot of Scala for a living -- just as a friendly (and intended to be a light hearted) riposte, some aspects of Scala strike me as "long in the tooth" too. With Scala 3 the language has done an admirable job to modernize but I find:
- The language feels heavy and (unnecessarily) "enterprise-y" -- reminiscent of the early 2000s rather than 2021
- The JVM is capable and performant, no doubt, but adds another heavy-weight and monolithic feel to the Scala platform. (Scala native likely to be essentially minuscule for years to come)
- The language veers towards a C++ style "I will have every PL feature." Sometimes less is more
- A Scala IDE (metals or JetBrains) feels clunky. sbt is over engineered and slow and given how important it is to Scala, does not give a good overall impression of the Scala platform
- Some questionable language features like implicits remind me of magic in Ruby (implicits are addressed in Scala 3 but I wonder how many years the ecosystem will have to deal with their complications -- forever??)
- The JVM seems to let down Scala in other places. Example (a) Null is rarely used in Scala but it could still pop-up in weird situations and not always because of Java interop. (Scala 3 tries to fix this via "explicit nulls" but there are compromises with that feature also). (b) A Functional style Scala (Cats and others) is popular. But true functional style has a lot of recursion. This, according to me, requires proper tail call support in the runtime which the JVM will never have. The Scala compiler tries to be smart but I wonder if it is able to deal with tail calls without blowing the stack in _all_ situations. In other words, it is difficult to do a "Haskell" on the JVM -- which we can see in a lot of places in the Scala ecosystem.
(BTW, I have pointed out some flaws of Scala but notwithstanding my criticism, Scala has got many good features that make it worthwhile. I may use it for a future project, lets see...)
> Having said that, Ocaml compiler is one of the greatest miracles in PL when it comes to speed vs complexity of the language.
I totally agree with the statement. Its a very balanced language in all important parameters: a high level of programming abstraction is possible, the LSP language server is responsive, the dune build system is great, compile times are really miniscule and run-time performance is great for a garbage collected language.
Do you still feel that way with Scala 3? From what I understood, the work on the DOT calculus helped reduce and simplify the core of the language.
Unfortunately I don't know much more about this other than "this is a Good Thing" and has helped/will help with dealing with edge cases in the language and the compiler, better type inference etc.
But Scala 3 still is overwhelmingly compatible with Scala 2.x (which is a required because of the tons of legacy code out there). Given that Scala 3 continues to be essentially the same language as Scala 2, the overall complexity of the language has not gone down very much even though the core of the language is now more consistent.
Put another way, the emergent complexity of the (tad more uniform) building blocks of Scala 3 still needs to be tackled by programmers.
I also want to point out that Scala 3 compilation speed is supposed to be faster but generally speaking the compiler is still slow-ish.
All in all, Scala 3 is more compelling than before. I may still adopt it in the future for a project. But I'm not as starry eyed about it than some others may be...
Python let's me start programming quickly. ML let's me finish quickly.