At least in my case, the point being made when I say "Just remember it's not out of altruism" is to remember that it's not out of altruism.