Do what makes you happy, but don't pretend that reading "difficult" books makes you morally-superior to the rest of us. I read fiction for fun, and non-fiction for personal growth.
You could read fiction for personal growth and non-fiction for fun as well.
Sure, there is the middle part ("books are supposed to be fun and the world is so awful why would you want to suffer"), but I took the bit about negative judgment to be the main thing the author was complaining about.