Hacker News Reader

Comment by compumike | Hacker News Reader

Also for a practical tip on YJIT memory usage, note that there is a "--yjit-exec-mem-size" option, see https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#co... for more details. (This command-line argument is mentioned in the paper https://dl.acm.org/doi/10.1145/3617651.3622982 but not in this blog post about the paper.)

At Heii On-Call https://heiioncall.com/ we use:

    ENV RUBY_YJIT_ENABLE=1                                                                                                             
    ENV RUBYOPT=--yjit-exec-mem-size=16

in our Dockerfile for our Rails processes.

View on HN

Wow, that's interesting and it seems a little crazy? From the docs:

    When JIT code size (RubyVM::YJIT.runtime_stats[:code_region_size]) 
    reaches this value, YJIT triggers "code GC" that frees all JIT 
    code and starts recompiling everything. Compiling code takes 
    some time, so scheduling code GC too frequently slows down your 
    application. Increasing --yjit-exec-mem-size may speed up your 
    application if RubyVM::YJIT.runtime_stats[:code_gc_count] is 
    not 0 or 1.

https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#co...

It just dumps all the JIT-compiled code? I'd expect to see some kind of heuristic or algorithm there... LFU or something.

The internals of a JIT are essentially black magic to me, and I know the people working on YJIT are super talented, so I am sure there is a good reason why they just dump everything instead of the least-frequently used stuff. Maybe the overhead of trying frecency outweighs the gains, maybe they just haven't implemented it yet, or maybe it's just a rarely-reached condition.

(I hope a YJIT team member sees this, I'm super curious now)

I don't work on YJIT but I _think_ i know the (or maybe an) answer to this. The code for a JIT'd ruby method isn't contiguous in one location in memory. When a ruby method is first compiled, a straightline path through the method is emitted , and branches are emitted as stub code. When the stub is hit, the incremental compilation of that branch then happens. I believe this is called "lazy basic block versioning".

When the stub is hit the code that gets generated is somewhere _else_ in executable memory, not contiguous with the original bit of the method. Because these "lazy basic blocks" are actually quite small, the bookkeeping involved in "where is the code for this ruby method" would actually be an appreciable fraction of the code size itself. Plus you then have to do more bookkeeping to make sure the method you want to GC isn't referred to by the generated code in another method.

Since low memory usage is an important YJIT goal, I guess this tradeoff isn't worth it.

Maybe someone who knows this better will come along and correct me :)

Seems you hit the nail on the head. Thanks for the informative answer!

As @xerxes901 said, there's some major challenge in freeing just one method code as it's not necessarily contiguous, and also it's of very variable size so it would generate lots of fragmentation. The allocate would need to be much more complex too to compensate.

But the team reasoning is that compilation isn't that slow, and while the code is freed, the statistics that drives the compilation are kept, so most of the work is already done.

Also the assumption behind code GC is that applications may experience a "phase change" e.g. the hottests code path at time t1, may not be so hot at time t2. If this is true, then it can be advantageous to recompile the hottests paths once in a while.

But that assumption is a major subject of debate between myself and the YJIT team, hence why I requested a `--yjit-disable-code-gc` flag for experimentation, and in 3.3 code GC will actually be disabled by default.

Huh! Thank you, that's helpful and informative. Thanks and for your contributions.

It definitely feels like the sort of feature for which there's no universal "best" default. A lot of applications might have "phase changes" and a lot of applications might not.

I would think that long-running apps (like Rails apps) would generally fall into the latter category.

Any recollection on how you arrived at the --yjit-exec-mem-size value? We've been running YJIT in production for some time, but haven't looked into tuning this at all.

Not parent poster and do not have production YJIT experience. =)

My guess is that you would monitor `RubyVM::YJIT.runtime_stats[:code_region_size]` and/or `RubyVM::YJIT.runtime_stats[:code_gc_count]` so that you can get a feel for a reasonable value for your application, as well as know whether or not the "code GC" is running frequently.

https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#pe...

That’s exactly right. Our code_region_size levels off a bit under 8 MB, so we set the limit to 16. In practice we see code_gc_count stays at 0.

Will have to look into this. Thanks for the suggestion!

Off Topic, but I thought Heii On-Call was done on Crystal? Or did I mixed it with something else?

Reply on news.ycombinator.com