Also "hard" is difficult to quantify. Humans are odd and quirky, and so are the languages they create (with perhaps the exception of conlangs like Esperanto or computer languages, but even those inherit the idiosyncrasies of their creators - Perl is a good example). Every language has its weird and difficult aspects, some more than others.
Japanese is "hard" because it has a complex writing system, but (for an English speaker) the pronunciation and grammar aren't particularly difficult: in fact I'd argue that Japanese is probably easier to learn to pronounce for an Anglophone than, say, French, and the verbs are considerably easier too. And a Japanese speaker would have an advantage in understanding Chinese characters, even though the languages have no kinship. Having once taught English, I'd say it's no cakewalk either, even for speakers of other Germanic languages: the spelling system barely deserves the name "system" and English verbal phrases are downright weird (put up with == "tolerate"? How?).