I never interpreted "listen to your body" as "give in to all your body's whims", it's more like: if in your experience you feel better after doing X, then X is probably good for you.