When it comes to the writing system, a Chinese character can have one pronunciation (two in very rare exceptions.) A Japanese character can have ten takes on how to pronounce the Chinese reading (onyomi), and be hammered into another ten Japanese-native words and verbs (kunyomi), and six ways they can be pronounced in names that your IME will refuse to ever recognize (nanori). The problem of not knowing how to pronounce what you are reading is made far worse in Japanese. With the one nice exception that material designed for young children sometimes have the syllables written above the characters for reference (furigana.)
Next up you have the SOV writing style instead of SVO. So unless you enjoy thinking and speaking like Yoda, you get to flip entire sentences in your head. In real-time.
And then you have particles that can mean a hundred different things based on context. Even some native speakers tend to have no idea when to use 'wa' or 'ga', it just becomes instinctual after a lifetime of language use.
Next, there's the verb conjugations. "They're simple, there's only one polite form, a handful of plain forms, and the two irregular verb forms!" ... and then they add on about a thousand possible verb endings that imply different meanings (-tai = to want, -takereba = to not want, -zu ni = another negative form like -nai but slightly more polite, etc.) And then you get the fun of stacking them! -saserareru (to be forced to do something) + -zu ni -> -saserarezu. And then you can extend that to be in the passive form, or to have it occur in the past ... and then you can just outright fuse two verbs together to form new ones! Tsuku + Nukeru -> Tsukinukeru + all your verb endings above = aneurysm to decipher in real-time conversation.
Then you have the polite language system (keigo, or alternate ways to say just about everything based on who you are talking to) where you have to practice for years before you won't either sound like a school girl or offend someone.
There are other hard parts (regional dialects, classical forms, synonyms, etc), but since Chinese shares them, I omit them here.
The big red herring is the two syllabic alphabets, hiragana and katakana. Those are very easy. But they do become a nightmare from hell when you are faced with a script that has no kanji at all (like older video games.) The synonym problem, on a scale of 1 to 10, goes from a 12 to a 50.
When I look at Chinese ... the biggest negative it has over Japanese, is that it's intensely more difficult in spoken form than Japanese. It's very hard to hear and reproduce the tonal marks correctly, yet that can completely change the meaning of words. And with so many similar sounds, you end up with poems like this: http://en.wikipedia.org/wiki/Lion-Eating_Poet_in_the_Stone_D...
Despite a lot of differences between how the Chinese and Japanese use the same characters, and in fact characters that only exist in one or the other ... horrifyingly, I find that I understand more when playing a Chinese game than a Japanese game, despite having studied Chinese for less than 1/20th as much as I have studied Japanese.