It's alright. Human languages aren't really logically tight the way computer languages are.

An example that goes completely unremarked on is "near miss", which logically means something that came close to missing but actually hit, and yet in idiomatic use means the opposite. People also get upset at "literally" to mean "figuratively", another one I find strange because it's an intensifier.

Clarity matters more in formal writing, and "couldn't care less" isn't particularly formal in any case.