There are many explanations for not spending time doing things you like; competing priorities is the main one (i.e. I like other things more than them) and also environment (perhaps you enjoy kayaking but hate the drive to get there, or don't want to go with johnny and max because you don't like max).
Just because you don't do something doesn't mean you don't like it. It's a false dichotomy.