Yes, it is bad. The distinction between less and fewer can aid interpretation of a sentence, especially in spoken language.

Consider: "I have less chocolate" vs "I have fewer chocolates"

In the first case, even if you don't hear the correct ending to the word chocolate, it is clear that you are talking about an amorphous mass of chocolate. In the second case, it is clear that you have distinct chocolates.

Natural language works on multiple redundancy, expecting some wastage as the information is conveyed. Any common mistake that detracts from the precision with which you can express yourself is a bad thing.

Less is analog, fewer is digital.