Hacker News Reader

Comment by tinco | Hacker News Reader

Crazy that she basically implemented a fully functioning JIT compiler that performs on par with the established commercial Javascript interpreters out there on many fronts in only two years using novel techniques, but reviewers reject her papers because they are unconvinced the novel technique has advantages over existing techniques? How does that even matter?

If someone discovers a new way of implementing compilers efficiently, then just that has scientific merit. Whether the new technique has different performance characteristics surely is simply ground for further research?

Anyway, awesome that at least the engineering community is very interested in this work of hers.

edit: I noticed the compiler is written in D, awesome choice of tech!

View on HN

Unfortunate as it is, peer review is quite broken and it is not quite clear how to fix it. Even if the intended goal is otherwise, in practice, peer review in CS turns into an adversarial exercise, with some differences in levels.

Rather than inspecting a submission and thinking if there is anything in the paper that might be of interest to the community, could the paper spur real progress, the review process usually boils down to: can I quickly skim to find something / anything with which I can shoot down this paper and get away with it and get done with this review. This of course is a generalization, not implying that this is what happened here. Unfortunately, the prototypical reviewer has become more like a prosecutor. There are various reasons why this has become the norm, people are aware of it, some are even trying hard to figure out ways to make it better, whereas some believe this is exactly how it should be.

If accuracy of prediction was a concern, Copernican model would never have been accepted. It took years of polish to make the then new model attain the accuracy of the then prevalent model. Some would argue that such models (the Copernican for example) should remain unpublished and under covers till they beat state of the art. I personally do not agree with this view. I would rather have a reasonably well baked but not yet perfect model out in the ether soon so that the model can enlist / recruit other people to work on it. It really boils down to where in the spectrum should the community operate: (i) spammed by advances of questionable merit, with the community spending effort to winnow the good from the bad or (ii) making sure that only the rock solid ones get through, at the risk of delaying or extinguishing useful advance and waiting for a rediscovery.

I'm guessing the paper was submitted to PLDI or POPL (wordpress is blocked here, so I can't read the context), the tier one PL conferences. The communities are not adversarial so much as they are enamored with science. I think from that perspective, rejection is understandable: they want more numbers and comparisons, and might accept the paper even if these numbers are not very enlightening (call it going through the motions). You also don't get many points just for wild experimentation.

For the same reason, many of the papers accepted seem very incremental and disappointing from a "new idea" perspective; and once you get the paper in, you have to go to the conference (I'm not a PLDI/POPL person)! One can always publish in a "new idea" PL conference (like my personal choice, Onward!, and there is even a new one called SNAPL from the people that brought us PLDI and POPL), but the points you get for publishing at those venues aren't as much as the true technical conferences (I don't care, since I have a job, but for a grad student, its a big deal).

My advice would be to suck it up and write a good PLDI paper. Be very thorough and honest with numbers and comparisons, and try to do it "right" (no pointless numbers that don't contribute to the story even if they might "work"). Then with a little bit of luck in PC reviewer selection (most PLDI reviewers are reasonable, some are not), the paper would probably get in. And if they do it right, it could be a really good paper that are lacking these days. Actually, that is another point: sometimes rejection is good, because the paper that gets in later is much much better. Patience.

This, I think, is a major weakness of the typical conference publication process, which is one-pass reject or accept. Journals have a much more humane process. Take the potential sentence from a reviewer:

"This work has significant novelty, but is lacking a comparison against technique XYZ."

For a Tier 1 conference submission, that's a rejection. For a journal, that's a revision request. Of course, in computer science, conferences are where the action is. But, there are attempts to split the difference. VLDB has moved to a more journal acceptance process (http://www.vldb.org/2015/submission-guidelines.html; rolling deadline at the start of each month through the year, reviewers can request revisions), which myself and some co-authors are currently going through. It is much more reasonable.

SIGPLAN is a conference-oriented community where the journals are not very interesting :). They also seem to be playing with more shepherding, but in my experience, once a paper has been tentatively accepted, it is pretty much going to get in unless the authors royally screw it up. There are also papers that are never going to make it in no matter what, and rolling deadlines can provide some false sense of hope there.

Concerning your critique of the in your opinion overly hard scientific review process. According to her blog the paper was rejected because: "Reviewers at conferences [...] have been very skeptical, and pressed us multiple times to produce some kind of comparison of basic block versioning against tracing."[1] I think this is a very valid concern. When you describe a new scientific approach (Basic Block Versioning) you should compare to the state of the art. Otherwise it is very hard for the reader to judge the merit of the new approach. However I agree with your sentiment that a new approach should not only be judged by performance numbers (there is actually a nice article going deeper on this topic on databasearchitects[2]). But there should at least be a more thorough theoretical discussion than only 3-4 sentences. Benchmarks comparing to existing approaches in this case can help to show pathological cases which might indicate weaknesses of the approach or to empirically demonstrate the feasibility of the new approach.

[1] http://pointersgonewild.wordpress.com/2014/11/14/the-fastest...

[2] http://databasearchitects.blogspot.de/2014/09/experiments-hu...

EDIT: Many conferences also allow publishing papers without a deep comparison to exisiting research in the industrial session. This allows demonstrations of interesting implementation variants or system choices

EDIT 2: The review critisim in older blog posts "Conference reviewers criticized us for not discussing compilation times, and raised the issue that perhaps basic block versioning could drastically increase compilation times." is also very valid for a jit compiler. Again discussion does not have to mean that you have to be faster than all existing systems. Paper acceptance is always a little bit a random process, but at least in this case the review comments are valid from my point of view and her phd advisor should probably have detected these problems in proofreading before submitting. I really hope that the paper will finally be accepted!

Note she's only comparing it with TraceMonkey (which is now gone from SpiderMonkey) and the initial V8 compiler (i.e., not the optimising Crankshaft compiler). This makes competing much easier. It's about the timespan I'd have expected — Carakan (the second latest in-browser JS VM; Chakra is the latest, but relatively little is known about it) was essentially feature complete and competitive on perf after nine months of development (with 3.5 developers, and an increasing number of QA from 0.5 up to 2). Carakan's big thing was finding and fixing ever more obscure bugs (especially in the compiler), taking almost the same time again.

Could it be that this technique is not as broadly applicable as the existing JIT techniques? Would it be possible to use the approach taken with Higgs to implement a language with a more open ended object system such as ruby? In JavaScript there's an upper bound on what a + b could mean, does BBV work well for more general method dispatch?

No reason it wouldn't work for ruby. It's about propagating type information. The way specific operators are implemented makes not much difference.

Reply on news.ycombinator.com