"Kannada has a rep of being one of the hardest languages in India."
This isn't really true. Kannada has a (relatively) regular structure.
Starting from scratch the most "difficult" of the four South Indian languages is probably Malayalam - the language of the state of Kerala. (note to hackers - the language name is a palindrome!)
It combines influences of Sanskrit and Tamil with a very unique pronounciation (to untrained ears Malayalam is more "flowing" and "liquid" vs the other South Indian languages which are more "staccato"). It is almost impossible to pick up the correct pronunciation and lilt of Malayalam without some deep immersion.
That said, I knew an Englishman,Graham Hall (now deceased unfortunately) a journalist/author who spoke fluent, accentless, perfect Malayalam. An incident where he chastised a local plumber trying to rip him off (no doubt assuming the blond foreigner was clueless about the local cost of repairs and prices of parts and such) in fluent Malayalam, with perfect slang, indicating what he thought of the plumbers' parentage and ancestry, made the local papers. (The thoroughly nonplussed and chastened plumber refused to take payment for his work and fled as soon as he could!)