I'm not sure what it's about. Maybe a holdover from French being the language of the English aristocracy during the middle ages?
As an American, my impression is that French is just considered more elegant and sophisticated than other foreign languages. I wouldn't consider it anymore difficult than another language, and quite a bit easier than some (e.g. Chinese).
...But I'm not a WASP, and if anyone has opinions similar to what you describe, my bet is they'd be WASPs.
I am not a historian and I'm quite happy to be corrected, but I'm suggesting these as factors.
It seems Romance Language academies love to squabble and love their "exclusive club". Though to be honest I haven't heard of a similar push to exclude people in other Academies
It’s just an english habit of bashing an idiotic gallicism, just like we find their queen concept surreal and idiotic ahah