Why don't compile Ruby using f.e. LLVM?
So why are Lisp and Smalltalk, which are equally dynamic and reflective, even faster?
There are tons of performance killers in Ruby, but most of them can be "worked around" by making assumptions about the common case and "just" spend some cycles verifying when the assumptions are violated.
E.g. 99.99% of the time, noone are stupid enough to redefine integer addition (because, besides the fact there aren't many good reasons to, all kinds of hell breaks loose if you do, for obvious reasons), so you can usually check the type tag on integers and inline the addition.
The problem is that edge case. But the edge case can be detected when someone tries to redefine '+' and you'd still save massive amounts of time if you were to set a flag at that point and still inline the additions, just with a test and branch to a slow path if people does something stupid (or in the case of JIT's, "just" re-jit everything at that point - yes, then overriding '+' will be slow, but common code will be much faster).
Gambit-C is one of the fastest Lisp implementations around, and it does have a full and performant implementations of call/cc. And I'm afraid that Common Lisp happens to be "fast by sccident" rather than "fast by design" - there's probably more extra speed to be gained by having a better implementable standard (which CL isn't, or at least not that much) than by omitting call/cc.
(Also note that call/cc is sort of out of favour these days, delimited continuations is where it's at.)
If you say that Common Lisp 'happens to be fast by accident', then this is completely misleading. 'fast' was a design goal and many implementors have done quite a lot to make demanding applications (like huge object-oriented CAD systems) to be fast.
It might have been one of the design goals, but it wasn't the ultimate design goal (which, I believe, was making a convergent compromise dialect to rule them all), otherwise certain parts of the language would have been most likely simpler. The things you mention are the good stuff. The problem is that there's also some bad stuff that hindered performance of CL on stock hardware until quality compilers were written. Yes, SBCL is quite fast. But you'd have a hard time trying to convince me that even modern implementations of Common Lisp represent some sort of performance optimum in the same way that, say, Stalin Scheme represents an optimum.
Also, wouldn't TAGBODY/GO be replaceable in presence of proper tail recursion by structuring the "TAGBODY basic blocks" into mutually recursive function calls? The same argument could be applied to inlining: Yes, the Common Lisp standard mentions inlining. But did you notice how nowadays, that's more or less considered an implementation issue rather than a language issue? How JVMs are capable of inlining just fine without having to be hinted what to inline and what not to inline?
Now I'd understand some of the hints, perhaps DYNAMIC-EXTENT makes somewhat more sense if the purpose is to say "It's my intent to create this value for a limited period of time, yell at me if I pass it out", but in general, from where I stand, a lot of the design decisions seem to make writing Common Lisp compilers actually more complicated to modern compiler writers.
> ultimate design goal (which, I believe, was making a convergent compromise dialect to rule them all), otherwise certain parts of the language would have been most likely simpler.
No, there was no ultimate design goal and Common Lisp was not designed as a compromise dialect at all. Common Lisp was designed to be a modern replacement for Maclisp and was incorporating several projects working on a modern Maclisp variant, for example NIL, Spicelisp and Zetalisp. Common Lisp was not designed to compromise over Emacs Lisp, Franz Lisp, Portable Standard Lisp, Lelisp, UCI Lisp, Interlisp, Scheme and whatever dialect was there at that time (Common Lisp first round of design was between 1982 and 1984. Then the standardization of ANSI CL was worked on until the early 90s.) If Common Lisp was compromise, then mostly between Maclisp successors. Later during standardization the Condition System of ZetaLisp was integrated, the LOOP of Maclisp and CLOS was developed as a new object system (based on New Flavors from Zetalisp and Portable Common Loops from Xerox). CLOS was also not as a compromise, but a radically new system optionally based on a Meta-Object System.
CLtL1 says: Common Lisp is a new dialect of Lisp, a successor to Maclisp. Influenced strongly by ZetaLisp and to some extent by Scheme and Interlisp.
> that hindered performance of CL on stock hardware until quality compilers were written
Quality compilers were written almost from day one: CMUCL, Allegro CL, LispWorks, Lucid CL. Of those Lucid CL was the fastest with the most optimizations.
Note that there are two or three different types of speed that needs to be addressed in CL: 1) the speed of unoptimized, fully dynamic and safe Lisp. 2) the speed of optimized, non-dynamic, and possibly unsafe Lisp. 3) the speed of production applications which need to be safe, somewhat dynamic, but optimized.
> Also, wouldn't TAGBODY/GO be replaceable in presence of proper tail recursion by structuring the "TAGBODY basic blocks" into mutually recursive function calls?
TCO is done by some implementations. Generally it is not seen as a useful default feature. Common Lisp favors iteration over TCO-based recursion. That's also what I favor. Since there are some implementation targets which don't support TCO (or where TCO makes implementations more difficult, one went to the simpler language). For example some of the Lisp Machines at that time did not support TCO. Today for example the JVM (where Common Lisp also runs on) does not support TCO. Scheme is seen as a different Lisp dialect and CL mostly learned from Scheme the default lexical binding - not more. Remember Scheme is much older than CL. CL did not adopt Scheme's one namespace, its calling semantics and conventions, not its naming conventions, not CALL/CC, not its macro system, ... This again shows that CL is not a compromise dialect - it simply does not support a lot Scheme-style programming - even though Scheme is a decade older than CL. By design.
> Yes, the Common Lisp standard mentions inlining. But did you notice how nowadays, that's more or less considered an implementation issue rather than a language issue? How JVMs are capable of inlining just fine without having to be hinted what to inline and what not to inline?
Common Lisp was not designed for one special implementation platform. Some CL compilers may honor INLINE declarations (many do), others do inlining automatically (like Allegro CL). If an implementation of Common Lisp on the JVM uses its inlining capabilities, fine. But there are many other implementations. ECL for example compiles to C and nobody is would say it is a good thing to stop working on that, just because the JVM exists. The JVM is a language issue. It's the Java Virtual Machine. It was not designed to host Lisp or support efficient implementations of Lisp. One can implement Lisp on the JVM, but it is a not so good fit.
> a lot of the design decisions seem to make writing Common Lisp compilers actually more complicated to modern compiler writers.
You are vague on these issues. Sure a good Common Lisp compiler is complex, but because the complexity is in things like the type system, type inferencing, etc etc. The base language itself is relatively simple. The result is that CL compilers still are producing faster code than most dynamic languages. Native implementations are still much faster and tighter than what the JVM can offer.
But even if it did: Rubyists are far more likely to be tolerant about documenting type information in optional comments than including them in the code.
I think there's room for experimenting with Ruby's typing, but I also think the most that project can hope to achieve would be extensions that could see some limited use in the odd little piece of performance critical code. It's best hope of success, actually, would be that JRuby and other alternative implementations grow more popular and make it harder for people to rely on C-extensions.
Furthermore, it appears to be a commercial product, yet searching for it reveals only a handful of link to their own site, and hardly no mentions elsewhere and the examples/docs are pitiful at best.
Looking quite shady to me.
the graphs are for startup times compared to JRuby
The first graph compares startup times. The second graph compares execution times. This has now been clarified on the site.
the examples/docs are pitiful at best
Working on it! What would you like the docs to address specifically?
Looking quite shady to me
I'm sorry you feel that way. Would you like a free evaluation license?
And the abuse of HTLAC patterns is due to the lack of non-HTLAC APIs; the syntax is trascurable
Method dispatch in Ruby, for example, can for the typical case (where nobody redefines random methods on all your classes on a regular basis once initial startup has been done) be little more expensive than a C++ virtual member function dispatch, but this requires work to either cache (and invalidate) method lookups, or pre-building vtables (and waste a bit of memory as well as being prepared to propagate updates to them if methods are redefined) for known methods.
Inlining of code is also possible, for example, but requires more work to add "guards" to protect against less common patterns that are still valid.
99% of Ruby is reasonably simple to optimise well. The problem is that the 1% can change the whole world for you and force you to either be able to selectively detect what changes are made or throw out all your optimisations), and so to get fast Ruby implementations will either need to ask for guarantees from the developer (pragma's to promise you'll never alter Fixnum, even in eval() blocks, or flags to by default freeze all standard classes, for example), or they'll need to start employing all the dirty tricks in the book.
Thankfully, the Smalltalk, Lisp, Self and Javascript worlds, amongst others, have pioneered plenty of dirty tricks that'll do just fine (e.g. consider Polymorphic Inline Caches, pioneered in Self, I believe, that are now quite widely applied for dynamic languages)
I think we'll eventually head towards a bit of both: Optional ways of making the job easier, such as promising not to much with things we normally don't muck with anyway, combined with new meta programming libraries to remove the reasons for a lot of the eval() usage, that will allow implementations to make much better judgements about what can actually change (instead of assuming all bets are off after an eval()), and a large number of dirty tricks (in the form of caches + heuristics + guards in various variations, to be able to take fast paths most of the time and "just" fall back to slow code for pathological cases).