The assumption is that if you wouldn't be reading, you could do something else that you would want to do. But if you are reading you can't. If you have more time than you know what to do with, then the argument doesn't hold.