It's still a "contresens" (can't find the right word in English, literally counter to its meaning), and should absolutely be avoided for clarity.
Let's not just say that it's alright
An example that goes completely unremarked on is "near miss", which logically means something that came close to missing but actually hit, and yet in idiomatic use means the opposite. People also get upset at "literally" to mean "figuratively", another one I find strange because it's an intensifier.
Clarity matters more in formal writing, and "couldn't care less" isn't particularly formal in any case.
And I can't agree with you. As a non native speaker, I deeply appreciate people making an effort to use language correctly to transmit information. I call that being mindfull of your interlocutors.
But "near miss" is more a parsing ambiguity, if not a mere disagreement about grammar. People who think it is illogical seem to assume it is "nearly missing". But in actual usage it is more that "near miss" is like a "narrow miss" and a "far miss" is like a "wide miss", all encoding distance to the implied target/hit zone.