The thing I and many other people hate the most in any language that has it (German included) are cases. The rest is easy.
What bugs me with German inability to reliably form plurals and having adjective endings vary by determiner (though not difficult to learn, but just seemingly serving no useful purpose).
For me I found aspect in Russia the most difficult thing grammatically to grasp. I hear learning articles from an "articleless" language is similarly difficult.
As a native German speaker, this is one of the few complaints about the German language that I can relive. This is really ugly.
Yeah I know, languages, organic, beautiful, but subtle things , etc etc.. But still.