Not really. You're underestimating how poor many people are. Even transport to a library can be a problem for some. And reading itself is a metabolic activity that takes work.
The poor obviously do read, but wealthy people have significantly more time and energy for the hobby, meaning that they read more.