When I hear someone say it, I often respond that "I could care less, but only a little bit" to make the point about that phrase.