Why would the majority of gems have anything at all to
do with Rails or need it to work as advertised?
If you put Rails-specific functionality in your gem, now you have a dependency on Rails, and given the seamless and transparent ways in which Rails (specifically, ActiveRecord and ActiveSupport) extend Ruby, it's easy to unintentionally wind up making the
non- Rails specific bits of your gem depend on Rails somehow.
Additionally one could say that AS/AR serve as sort of a defacto standard library of sorts at this point. You copy some code from Stack Overflow, it's got a dependency on one or the other, etc.
I'm not endorsing that state of affairs.
I actually don't know how I even feel about that state of affairs. The mostly-transparent extensibility of Ruby is the best and the worst thing about it. It's great if you're writing a framework or gem; but it makes it really tough for beginners to know what is doing what.