I am just trying to understand your case. So if you have a Gemfile in your folder and then execute something like
bundle config set --local path "vendor/bundle"
Then all gems should be installed in ./vendor/bundle.I am not sure how Passenger works (as I am mostly using Puma). I am assuming based on what you wrote that the issue was with the gem passenger itself which is used by the Apache config and it is not a library used by your project.
I don't think the issue you encounter is representative to Ruby and it's ecosystem. I think any module loaded to Apache will probably need a way to know how to load some other modules and probably passenger (via Apache config) needs to know where to find the dependencies it has. Passenger is a web-server so it has its own dependencies.
My suggestion was about the dependencies of your own project.
I am usually doing projects with Rails and Sinatra + Puma + Nginx or Apache and in both of them with the setup I shared I did not encountered any problems releasing the project in production with bare metal configuration (via Capistrano for example) or using Docker.
I am not saying that my experience is representative for everything in Ruby ecosystem. I started using Ruby in 2007 with Ruby 1.8 and I do agree that we had some rough years in the ecosystem specially when working on multiple projects simultaneously.
But now, my experience is a breeze.
On a tangent, I have more problems with npm/yarn than with gems. For example I just started to upgrade an old Rails app to Rails 7 and suddenly on my Mac OS a npm package needs some kind of Python version with some libraries that in turn seems to require some other stuff. Even if I am more like a dinosaurs and prefer to run locally all projects (thank maintainers for asdf) for this one I let it go and put it in docker :)