Extension Points, or how OCaml is becoming more like Lisp | Hacker News Reader

Extension Points, or how OCaml is becoming more like Lisp

blogs.janestreet.com

104 pointsedwintorok29 comments

One thing that's not nice in typical Lisp macros: for really complex transformations one needs a code walker. Unfortunately that part has a few problems: no standard code walker, complex api, different ways to access crucial information about the code in various implementations, ...

OCaml has a type-safe AST walker (two actually - one is part of camlp4 and the other called ocaml-ast-analyze).

It would be interesting to see how the speed of -ppx extensions compares to camlp4. If camlp4 is run in bytecode mode it can be quite slow, I looked at 'top' while running 'opam install core' at some point, and most of the time was taken by camlp4. Since -ppx extensions are supposed to run natively (without requiring natdynlink support even) I would expect them to be faster.

Has anyone done any comparisons?

There are pros and cons to not using dynlink, since it requires another fork (although this can be avoided using a custom compiler driver).

Avoiding bytecode in the compilation toolchain is really the main benefit though. This was originally due to the fact that only the bytecode backend supported dynamic loading (which is used by camlp4 to load the syntax extension modules). When native dynamic code loading was added to OCaml circa 2009 or so, the build systems around camlp4 never really took up the change to switch to the much faster native code version. All the ppx extensions emerging now are at least guaranteed to have a native code version on all the major supported architectures (x86, ARM, Sparc, PPC)

Cool. Language extensions are easier to write. Honestly, never had the guts to really learn camlp4 beyond an introductory tutorial, so I am happy to hear that this new approach is easier and also more practical.

There is a link from the article with a more complete example of how the extension points really work: http://whitequark.org/blog/2014/04/16/a-guide-to-extension-p...

OCaml is great because it is doing a lot of things right. Hope, this new feature will become more widely used than camlp4, and become a stable feature of the language.

As languages improve, they keep moving towards Lisp — a story that has been playing out for decades. There is a reason Lisp is called "the final language".

Only if you're looking from a particular angle. Given the growth of more and more powerful type systems I'd say there's an important sense in which, as languages improve, they keep moving away from Lisp.

This story isn't really OCaml becoming more like Lisp; rather, it's OCaml finding a way to implement Lisp's best feature without the downsides of being Lisp.

Only if you're looking from a particular angle

I'd say most angles except the type system as you point out, or am I missing something?

I think many languages are realizing that syntactic abstraction is important and thus move toward lisp. I think many languages are realizing that HOF are important and are moving toward lambda calculus (which is simultaneously motion toward lisp). I think languages are learning that modules and the abstraction embedded in them is important and thus are moving toward ML (slowly). I think languages are learning that concurrency via isolated actors is a pretty good way to manage state in a concurrent system and thus are moving toward CSP (which is simultaneous motion toward lisp). Even more on that last point they might be realizing that process trees are a great way to make stable concurrent systems and thus are moving toward OTP.

Clojure is arguably a better language for being less pure as a Lisp. Many old school Lispers object to the dedicated vector and map syntax, but it's 2014 and these data structures are important and common enough to warrant shorthand.

Syntax is a trade between regularity and expressiveness. What you're seeing with "languages improving towards Lisp" is, in my opinion, languages seeking the benefits of greater regularity.

Old school Lispers don't object to vector or map syntax. Actually CL has vector syntax #(v e c t o r) and easily can add other data type syntax - that's why reader macros exist.

I don't know. Syntax is... hard to judge, but there's certainly some momentum behind the idea of a more rigid syntax (e.g. python significant whitespace, go shipping with formatting tools) that belongs more to the Fortran tradition than the Lisp one. OO style seems to be coming out of the other end of the hype cycle, and while many argue that OO is a lispy concept I don't think the community ever really got behind it. Metaprogramming is still an open field, but I see as much going on with languages that make a strict delineation between compile and runtime as with the looser lisp approach.

I think Lisp and Scheme have excellent macro systems, which are eminently worth stealing. I don't think OCaml will evolve into them on the type-system level though...

The thing about the OP is that Lisp isn't well defined. The only constant is that what Lisp is changes, so it's really not that useful to talk about Lisp as "the final language" because there will always be new Lisps. The only common thing between the Lisps seems to be the s-expressions and metaprogramming.

Dialects evolve, sure. Although it's worth noting that if one had adopted Common Lisp in the '90s one would have a stable base that is today, 20 years later, still ahead of basically everything on the language level (except for the type system as noted in other comments).

Shen lisp has the world's only turing complete type system (making it even more advanced than Haskell's system).

It's not the only one by any means. E.g. scala's type system is turing complete - http://michid.wordpress.com/2010/01/29/scala-type-level-enco...

Do you really want a turing complete type system? Being turing complete does not necessarily mean more powerful for real world applications.

You should check out some dependently typed languages.

With the downside of undecidable type inference and checking. Haskell's type system really hits the sweet spot between power and decidability.

Thank you for Shen Lisp, I wasn't aware of it.

The downside of the phrase Turing complete is that there isn't a practical limitation that instantly comes to mind when you hear that phrase. You don't hear people say "Oh no, that wouldn't be Turing complete". Complete or not, how does it affect a real application?

For a type system, being Turing complete is a disadvantage: you can't prove that your typechecker will always terminate.

The result is even stronger: we can prove that it will sometimes not terminate!

Still that's not too bad, since type checking is always conservative: if a program passes, we know it's correct(ly typed). If it doesn't pass, we don't gain any knowledge: it may be incorrect, or it may be correct in a way which the type checker's limited algorithm cannot determine (ie. a Goedel sentence).

I'm very concerned whether my programs terminate/coterminate, since having to kill them part-way-through could cause corruption and other nastiness. Whether the type-checker terminates or not I don't really care about; I can just kill it after a certain timeout and keep fiddling with my code until it passes, just like any other type error.

Of course, a timeout removes Turing-completeness, but that timeout is under my control at the commandline, rather than being an inherent property of the algorithm.

This is only partially true. The type system is only unable to prove termination if you choose to use those constructs which may disallow termination.

I don't know about you but I'd rather not have my type checker hang and force me to interrupt the process all the time.

I'm not convinced by the example given at the end of the article. For this precise very simple use case it might be ok, but what about more complicated syntax extensions?

Camlp4 is still very much around and maintained out of tree, and will work in the next release of OCaml (https://github.com/ocaml/camlp4). Extension points are intended to be simpler and more easily integrated into editors for the common case, but you can still use camlp4 for extensions that genuinely extend the OCaml grammar.

Interesting, seem similar to annotations/attributes. Also some similarity with AST handling in other languages.

Now I have something to play with on the upcoming weekend. :)

been writing OCaml since years. Def good news

Reply on news.ycombinator.com