Hacker News Reader

Comment by sundarurfriend | Hacker News Reader

So with the promised 3x speed improvement (hooray!), where does Ruby stand with regard to performance, compared to say Python? According to [this benchmark][1], it seems (the 3.0.0-preview1 version was) on average about 2x slower than Python. Are there other benchmarks to compare with?

Looking into it, this other benchmark [2] of 3.0.0-preview1 vs Ruby 2.7 actually concludes there's no real performance improvement there. So I guess an additional question is, is there actually a 3x speed increase in realistic benchmarks?

[1] https://benchmarksgame-team.pages.debian.net/benchmarksgame/... [2] https://www.fastruby.io/blog/rails/performance/ruby/hows-the...

View on HN

> where does Ruby stand with regard to performance, compared to say Python?

In my experience Ruby has been faster than Python for a long time - ever since its performance improved significantly going from 1.8 to 1.9, and Python’s got worse going from 2.x to 3.x.

Python 3.6 was pretty much on par with 2.7 and that was 5 years ago. There doesn't seem to be any meaningful difference between python and ruby these days at all https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

I did some measurements recently with 3.0.0 RC1 compared to 2.1.0 based on https://github.com/smarr/are-we-fast-yet. Here are the results: http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub.... Ruby 3.0 without JIT is factor 1.2 faster than Ruby 2.1.0; Ruby 3.0 with JIT is only factor 1.4 faster than Ruby 2.1.0. Crystal is about factor 20 faster.

Interesting results, thanks for sharing, that matches my expectations. OptCarrot is probably one of the rare cases where Ruby 3.0+JIT is 3x Ruby 2.0. On most benchmarks, the gains are much smaller.

I'll try to run these benchmarks on TruffleRuby (and maybe JRuby) too, would be an interesting comparison.

BTW, is there any reason you used 2.1.0 and not 2.0.0? Maybe some issue with 2.0.0?

> OptCarrot is probably one of the rare cases where Ruby 3.0+JIT is 3x Ruby 2.0

The Are-we-fast-yet suite is optimized for inter-language comparisons and sets the focus on ideomatic language use and representative algorithms. There is a peer-reviewed scientific publication; here is a preprint: https://stefan-marr.de/papers/dls-marr-et-al-cross-language-...

> is there any reason you used 2.1.0 and not 2.0.0?

The 2.0 version produced errors like "uninitialized constant Process::CLOCK_MONOTONIC (NameError)"; from that I concluded that it is not compatible. I still had to apply a fix (https://bugs.ruby-lang.org/issues/9578) to make Ruby 2.1.0 compile on my test machine. Here is a report version where I compare with Ruby 2.7 instead of 2.1: http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub...

> The Are-we-fast-yet suite is optimized for inter-language comparisons and sets the focus on ideomatic language use and representative algorithms. There is a peer-reviewed scientific publication; here is a preprint: https://stefan-marr.de/papers/dls-marr-et-al-cross-language-...

Benoit (eregon) is the second author of that paper.

Well, apparently he does not want to be recognized; without your hint I would not have figured it out (there is also no indication in the profile). At least I don't have to convince him of the benchmark's quality.

EDIT: surprised to learn that his group is responsible for the GraalVM and even located in Zurich (https://labs.oracle.com/pls/apex/f?p=LABS:project_details:0:...) which is not far from the place I live; it's a small world ;-)

the Ruby 2.7 benchmarks [0] seem to shows that Ruby is slightly faster than Python 3, or at least fairly competitive.

Though things like Python 3 going from 18.45 seconds to 2.67 seconds on the regex-redux test [1][2] tends to make me question the validity/usefulness of these benchmarks. (inasmuch - how much of it is testing language performance vs bypassing it?)

[0] https://web.archive.org/web/20200301005643/https://benchmark... [1] https://web.archive.org/web/20191225153009/https://benchmark... [2] https://web.archive.org/web/20200604134902/https://benchmark...

Ruby version of regex-redux uses built-in regex support, and I'm not even sure if it uses it the most performant way.

Significant portion of the Python test case is FFI code to directly link PCRE bypassing even normal Python extension support.

I guess I should augment some of the SBCL benchmarks with custom assembly then...

> …SBCL benchmarks with custom assembly then…

How could that possibly effect the performance of the Ruby regex-redux? :-)

"Wanted — Ruby PCRE2 regex-redux program"

https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...

It wouldn't help ruby, but I'd rather write common lisp ;)

Fair enough :-)

(Although they wouldn't be accepted — the option of using those widely libraries is only for pi-digits and regex-redux; and not "custom assembly".)

I've also found that the Computer Language Benchmarks Game benchmarks tend to be less and less representative of real workloads, due to what the benchmarks do and to the philosophy of "write it to make it fast, not idiomatic, and parallelize things that would typically not be parallelized". Just taking a look at the benchmark code makes it clear it's very far from idiomatic Ruby code.

I think it's unfortunate that you've chosen to make such vague disparaging comments.

Here are 37 Ruby programs — https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Presumbly you can show us 2 lists: which of them are your idea of "idiomatic" and which are not ?

Merry Christmas.

> …the philosophy of…

Actual quotation —

'You will probably come across people saying that the programs are not idiomatic ("enough"). So read the description and write your own idiomatic program, without programming tricks.'

https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...

> Python 3 going from 18.45 seconds to 2.67 seconds on the regex-redux

You do realize those are different programs, don't you?

You do realize that both those Python programs are shown in the current measurements, don't you?

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

But if you really want to show us the old Q6600 measurements so many people have complained about, the latest version is here:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

> 2x slower than Python

Several multi-core Ruby programs failed when run with 3.0.0-preview1 --jit

Which is why you see Python3 programs using 4 cores for spectral-norm and binary-trees and fannkuch-redux — when the corresponding Ruby program is only using one core.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

It’s multi-core age today. Concurrency is very important. With Ractor, along with Async Fiber, Ruby will be a real concurrent language. — Matz

We need new Ruby programs for the benchmarks game, to show that new concurrent language.

If you’re interested in speed, check this repo: even among Ruby idioms there are big speed differences.

https://github.com/JuanitoFatas/fast-ruby

The promised gains were (basically) achieved:

1. on the optcarrot benchmark 2. between Ruby 2.0.0 and Ruby 3.0.0

The speed difference between 2.7 and 3.0 is completely moot.

The few times I tested, Ruby 2 was ~2x faster than Python 3 in tasks involving iterations on long lists.

Reply on news.ycombinator.com