What really irks me is that in case-insensitive languages, codebases naturally start to use all kinds of variations of casing, to refer to the same variable. It really trips me up when reading code. It's also difficult when writing code since I tend to remember variable names in a very visual way. (Same reason why I dislike the case-insensitive Windows filesystems).
There simply isn't any good reason not to always use the same case for any variable throughout a project.
Beyond that it's also aesthetically not pleasing, from an engineering viewpoint. Case-insensitive comparison is much more complicated than simply comparing strings as arrays of bytes.
Historically, I think the reason why case-insensitive languages exist is simply that at the time they were invented (at least in the case of Pascal), many systems couldn't input lowercase characters.