Same here. I think the reason it bugs me so much is because my mind views it as a type error:
int fewer;
float less;
is how I tend to remember the distinction. (Fortunately in my family, my wife is the language nerd who first called me on my misuse many years ago. Now I notice it everywhere too. It's weirdly viral.)