Hacker News Reader

Comment by kbd | Hacker News Reader

Congrats to the PyPy team on what sounds like a pretty big release!

Something in the release notes caught my eye:

> The past months have seen pypy mature and grow, as rpython becomes the goto solution for writing fast dynamic language interpreters.

I asked this question[1] on the Perl 6 thread from a few days ago but didn't get an answer. Does anyone know why on earth the Perl 6 folks created yet another dynamic language VM+JIT with MoarVM instead of taking advantage of all the great work done with PyPy? Does anyone know whether PyPy was even considered as a target before writing MoarVM?

[1] https://news.ycombinator.com/item?id=8982229

View on HN

I tried talking to them, at the time they were sold on using JVM (noone got fired for using JVM for your VM). Too much hype, too little cold thought

I remember that conversation. The lead developer of the project had C# and Java experience and had already written his prototypes there.

I remember one of the times you dropped into their IRC channel to ask (not sure if there were more times). I wasn't really thrilled with the way that was handled. AIUI, they have reasons why PyPy isn't considered the best choice for Rakudo, they just did an exceedingly poor job of communicating them.

Perl6 started long before PyPy (2000 vs ~2007).

And yet, work began on MoarVM only in 2012.

http://en.wikipedia.org/wiki/MoarVM

I think MOAR was designed specifically to run NQP efficiently.

NQP is a subset of Perl 6 which the Perl 6 compiler is written in. That compiler takes a program written in Perl 6 and converts it into NQP. That NQP program is then compiled to different targets such as JVM, Parrot etc. This is my understanding, of course.

Sounds like someone needs to write an NQP JIT in rpython.

They would have their work cut out for them, that I can see. They could reuse the .moarvm bytecode and file format, the NQP compiler and Moar backend [0]. I think many of the test cases are even common between the three backends... So all that's left to re-implement is a bytecode interpreter, IO facilities and FFI. That is, if my understanding of MoarVM is accurate.

[0] (https://github.com/perl6/nqp/tree/master/src/vm)

I don't want to hijack a pypy thread with perl stuff, but this needs to be corrected:

> I think MOAR was designed specifically to run NQP efficiently.

yes

> NQP is a subset of Perl 6 which the Perl 6 compiler is written in.

yes

> That compiler takes a program written in Perl 6 and converts it into NQP.

No. moar is a vm backend for nqp, with a traditional bytecode interpreter, a jit, gc, ffi, and binary format.

>That NQP program is then compiled to different targets such as JVM, Parrot etc. This is my understanding, of course.

No. Of the current three perl6 backend moar, parrot and jvm is the fastest, but has problems with its traditional threading model. It does not scale linearily up the number of physical cores. It needs locks on all data structures, but this is still better than with perl5 or pypy. parrot does scale linearily, has locks only in the scheduler, but needs compiler support to create writer functions, only owner can write, access through automatically created proxies. The jvm threading model is also faster and well known, but has this huge startup overhead.

perl5 has to clone all active data on thread init and creates writer hooks to copy updates back to the owner. this is a huge startup overhead, similar to the jvm.

Overall, would you expect any c/perl compiler dev to switch over to rpython to back your perl6 vm? Writing a jit for a good vm was the matter of a summer, and writing the compiler optimizations needs to be done in the high-level language of choice to be more efficient and maintainable.

You should better look over to rperl to compare it to rpython. This is the similar approach. A fast restricted optimizable language subset, which compiles do typed C++, with fallbacks on unoptimizable types to the slow perl5 backend. No need for a jit with its compilation and memory overhead. A jit only makes sense for dynamic profile-guided optimizations, as in pypy or v8.

Even if the simple static jit's as in p2 or luajit without much compiler optimizations still run circles around those huge battleships in the jvm, clr, v8, pypy or moar. optimized data structures still are leading the game, not optimizing compilers.

rurban, I just want to point out that I was not calling for an rpython target for nqp nor was I claiming nqp is the rpython of the perl world. Not really sure what you are correcting.

>No. moar is a vm backend for nqp, with a traditional bytecode interpreter, a jit, gc, ffi, and binary format.

But surely NQP is a "language" and Moar is a vm ?

For the question, the start date of MoarVM is more interesting than the start date of Perl 6.

It's also not quite fair to start Perl 6 at announcement and PyPy at their first release (they were organized enough in 2004 to be applying for EU funds).

No, not really, as it ignores all the specifics of the situation which matter. The current leading implementation of Perl 6 is Rakudo, which is implemented in NQP (itself a subset of Perl 6 that's meant to be easier to implement and optimize), which is comparable in some ways to rpython. At the point you already have an intermediate language, a lot of the benefits of using PyPy and being able to write your language in a higher level language aren't as much of a gain. Additionally, using PyPy would lock the implementation into whatever current limitations PyPy has (threads?). MoarVM is not the back-end, it's one back-end, and one meant to optimize specifically for NQP's needs.

I imagine at some point a PyPy targeted Perl 6 will appear. It may or may not be based on the Rakudo implementation as a target for NQP (jnthn was able to get initial JVM support for NQP completed in a summer), or another implementation that targets PyPy through rperl directly. It would be really interesting to see what Perl 6's automatic opportunistic auto-threading of list operations and PyPy's STM work could do together.

That's a whole different (perfectly reasonable) line of reasoning that focuses on the needs of Perl 6 as an explanation.

The comment I replied to frames it such that PyPy wasn't an option for MoarVM because it didn't exist when things got started.

I don't know, but just from your description it sounds like a case of NIH[1].

[1] http://en.wikipedia.org/wiki/Not_invented_here

PyPy is crippled by the GIL, why would you base your next hot new language that aims to be mainstream on that?

You mean, like V8 being crippled by the GIL?

Why would you need a GIL if you don't have threads in the first place?

PyPy is actively working on an STM-based alternative. Does MoarVM avoid a GIL? If so, what approach are they taking?

Since Perl 5 doesn't have a GIL, I doubt any of the Perl 6s will. Threaded Perl has been a thing for 15 years or so. It's not something to "solve", either your interpreter is reentrant, or it isn't.

But spawning threads in your Perl code comes with the same set of problems as anywhere else, it's easy to create races or deadlocks, and Perl happily lets you do these things if you ask it to.

I think Perl 5 threading model is pretty broken. The official documentation explicitly discourages the use of threads [1]. Perl's threading model consists in running a separate interpreter on each thread without any data sharing, which practically the same as using multiple process except without the advantages.

[1] http://perldoc.perl.org/threads.html

It's not quite as bad as that, but it's close. There are some use cases where it's beneficial. The one I use it for is where I'm pre-creating workers and using the shared memory mechanisms and signalling of threads (which work quite well in Perl where it auto releases the lock based on scope).

The GIL is in PyPy, not in RPython, nothing precludes writing a GIL-less RPython-based VM.

However RPython is not thread-safe and the GC it provides is not thread-aware, so you'd have to provide your own thread-aware GC (or a big GC lock)

the stm-gc can be run without the STM in a way that's thread safe and does not have the STM overhead. It's almost as good as the PyPy GC (it's missing a few optimizations and it's not incremental, but can be made to work).

Reply on news.ycombinator.com