I think it comes mostly from CI environments that start entirely clean before every run. 10 minutes every time a commit is pushed is not pleasant. That's not how I'd like CI to work, but sadly it seems to be the current state of things.
I've only used circleci and github actions for this, but in both cases precompiled rubies are available. On circleci you run your tests on a ruby docker image with the right version of ruby installed. On Github Actions I use https://github.com/ruby/setup-ruby, which installs an compiled ruby in a second or two, and also properly caches your gems. I think if someone's CI environment is building ruby from source they are doing it wrong.