Benchmarking CRuby, MJIT, YJIT, JRuby, and TruffleRuby | Hacker News Reader

Benchmarking CRuby, MJIT, YJIT, JRuby, and TruffleRuby

eregon.me

91 pointsshelajev29 comments

The Truffleruby results look amazing but leave me with the question why we don’t see stories of implementation at major Ruby / Rails sites. Too soon, licensing or not yet 100% compatible?

Windows support?

Pretty sure Shopify uses it for some apps.

Thanks - yes of course Chris Seaton works at Shopify.

Still a bit puzzled why we haven't seen any headlines about major savings.

It isn't 100% MRI 3.0 compatible[0] and wasn't able to run Rails at first. It's also not always faster for Rails which occupies a lot of Ruby mindshare.

[0] https://github.com/oracle/truffleruby#current-status

Is the Rails performance something to do with the amount of meta programming going on - I can imagine that being hard to optimise!

Well YJIT succeeded to improve Rails by 30% in a year , so it IS possible. https://speed.yjit.org/

It's the same benchmark in the blog post, `railsbench`. So TruffleRuby already speeds up some Rails apps like that one but not every Rails app/program. (TruffleRuby 3.27x, YJIT 1.33x on railsbench)

I'm not sure. If Chris Seaton pops into the thread, you should ask him.

Actually TruffleRuby optimizes metaprogramming more than any other Ruby implementation. Rails is not the issue, big codebases is the issue: more code = more chances to use something which is not so fast on TruffleRuby yet = more chance to become a bottleneck and degrade the overall performance.

Thanks for the thoughtful and insightful responses. Understand that a large code base is hard - your point here though presumably means that some things in truffleruby are significantly slower than CRuby to cancel out the gains. Or have I misunderstood?!

Yes, some things are slower. Most of them I'd say are unexpected performance bugs. I'd think most are easy to fix once investigated, but some can be hard to fix (recent example, `File.read` is quite fast on CRuby).

Some things are expected to be slower, for instance constantly redefining (monkey-patching) methods or constants is slower on TruffleRuby, but that's typically because the program is broken and so it'd be slow on CRuby as well.

Thanks again and of course for Truffleruby!

I just ran a comparison of a benchmarking script we use for our own product and saw wildly different results. There's no way I'd consider migrating to truffle today:

# With MRI ruby-2.6.6

Measuring [loading ruby dependencies] 3.560960 2.507366 6.075755 ( 6.154898)

Measuring [loading configuration yaml] 0.823779 0.085605 0.909384 ( 0.931334)

Measuring [creating objects from YAML and performing filtering] 0.499576 0.042023 0.541599 ( 0.552427)

Measuring [object validation] 2.362139 0.226133 2.588272 ( 2.674958)

Total time: 7.246786 2.861230 10.115445 ( 10.314025)

# With truffleruby-21.3.0

Measuring [loading ruby dependencies] 91.818473 5.598149 97.427109 ( 36.676471)

Measuring [loading configuration yaml] 28.032630 0.811577 28.844207 ( 8.616437)

Measuring [creating objects from YAML and performing filtering] 14.952781 0.479387 15.432168 ( 4.634803)

Measuring [object validation] 86.747057 2.788266 89.535323 ( 28.324909)

Total time: 221.595872 9.680172 231.286531 ( 78.264226)

Startup is likely to be worse, because it typically runs a lot of different code for not long and the JIT might not have enough time to optimize much of that. OTOH the JIT needs to learn what the program is doing, i.e., profile it in a sense, and that has a cost on interpreter speed. Still these numbers are worse than expected, especially [object validation], so if there is a way to reproduce it'd be great if you can open an issue about it.

Persisting the JITed code is what we think can solve the slower startup entirely: https://www.graalvm.org/graalvm-as-a-platform/language-imple... Also other things mentioned in https://eregon.me/blog/2022/01/06/benchmarking-cruby-mjit-yj...

Many years ago (2011 or so?), I worked on a Rails app that used JRuby. The driving force was a small part of the app needing a Java library. I think we did see some performance benefit, or at least benefit from using multi-threaded web servers instead of process-per-request servers (like Unicorn).

However, we also ran into many problems that felt like we were the first to encounter them (few or no similar reports on project bug lists or StackOverflow etc.) which is never fun if you're under any kind of pressure to deliver.

In hindsight, if I had something small and reasonably isolated (a microservice?), it could make sense to use JRuby or Truffle (if you need some JVM lib; if it's purely for performance reasons I'd just grab a different language for that microservice). But for a larger app that pays the bills, I'd stick with the well-trod path (at the time that was MRI, Unicorn, single-threaded).

Glad to hear you got some benefit from JRuby! A lot has changed since 2011, including much better compatibility and many little performance improvements. I'm not sure we were even on the current JRuby runtime (register-based IR + JVM JIT) at that time, which has improved perf and compat tremendously.

If you ever have any issues with JRuby again, please let us know. We spend most of our time supporting users and want them to have a good experience.

Explanation from Stripe¹:

> Nearly all of Stripe’s codebase is implemented in Ruby running on the default Ruby VM (YARV). Not only did we not need Java VM-level interoperability, choosing either alternative Ruby implementation would have made for a difficult migration path. Stripe relies heavily on gems with native extensions, and as you can imagine, a multi-million line Ruby codebase over time starts to depend on Ruby-the-implementation, not just Ruby-the-language.

¹: https://sorbet.org/blog/2021/07/30/open-sourcing-sorbet-comp...

That second sentence seems a fair assessment. TruffleRuby does support many native (C/C++) extensions so that's rarely an issue. But indeed in such a large codebase it's likely to depend unexpectedly on CRuby-specific behavior. And while that can be fixed it takes some effort either in TruffleRuby (to match CRuby) or in the app (e.g. to avoid relying on `RubyVM`).

Compatibility is one, it's hard to be 100% compatible with CRuby, and large codebases tend to sometimes depend unintentionally on weird behavior or even bugs in CRuby.

Keeping up compatibility (while keeping things efficient) is a lot of work, TruffleRuby tries to reduce that by reusing as much as possible existing code, including reusing C extensions shipped with CRuby.

JRuby is in wide deployment at hundreds or thousands of businesses across every industry. We generally run real-world apps much faster than CRuby (lower latency, better use of CPU and memory) and are especially good at big data and high concurrency applications.

JRuby remains the only alternative Ruby to see large-scale production deployment.

I use a benchmark that's pretty rough on languages, basically lots of maths and loops.

Python 3 and Ruby 3 (with no jit) both take about 90 seconds on my computer.

Ruby 3.1 with --YJIT takes about 35 seconds.

TruffleRuby takes 9 seconds (!!)

JS (Chrome) takes 8 seconds.

C++ takes just over a second and C is about a second even.

Please use something "basically lots of maths and loops" where we can see the source code (like the blog post does).

n-body

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

spectral-norm

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

I keep forgetting just how fast TruffleRuby is.

What is the downside to TruffleRuby? Memory consumption since its JVM based?

Very subtle differences in behavior. It's fine if you start writing your project targeting TruffleRuby, but its a nightmare porting a massive existing ruby codebase.

Compatibility is certainly an issue for massive codebases, OTOH I think over time TruffleRuby is getting closer and closer to CRuby behavior to the point it would be fairly rare to find a compatibility issue. Do you have personal experience trying to run such a codebase on TruffleRuby?

We noticed many numerical operations sometimes didn't return the same class as cruby (Fixnum/Bignum/BigDecimal), sometimes resulting in wildly different calculation results.

Numerical to string formatting is a bit different too. Which makes the above problem even worse, compounded by how frequently ruby web code goes string to numbers to string to numbers.

Regexes behave differently if they worked at all.

When did you try? Fixnum/Bignum are gone since many years (before 1.0RC1 which was April 2018, it's all Integer now). So I guess many years ago, and back then TruffleRuby was basically implementing Ruby features, had fairly limited compatibility and likely could not run Rails, very different from today.

There were compatibility issues with BigDecimal, but TruffleRuby now uses the C extension, hence it should be exactly the same behavior as CRuby.

> Numerical to string formatting is a bit different too.

AFAIK that was fixed years ago if you mean float formatting.

> Regexes behave differently if they worked at all.

TruffleRuby always used Joni, which is literally a translation of CRuby's Regexp engine to Java (by the JRuby team), so that is very surprising and I have a really hard time to believe it. At least "Regexes behave differently if they worked at all" seems harsh and highly inaccurate to me. There likely were a couple Regexp issues but the generalization seems wildly exaggerated.

Reply on news.ycombinator.com