German definitely is difficult in some respects for an English speaker. English grammar has simplified over time in ways that German has not. For instance English used to have a distinction between formal and informal you (thou and ye), centuries ago but that long since passed into disuse. Nowadays there is only one form of "you". Likewise, the English of the 11th century had the Germanic three genders: male, female and neutral. However Old English would be unintelligible to a modern speaker. Gender was gone by the 1300's.
All these things cause tremendous problems for learning the language because you have to track distinctions in the language that don't exist natively so you never had to think about before. Native speakers learn genders just by hearing them repeatedly, at the same time as learning the words, but English speakers learning German normally need to study dictionaries a lot to learn new words and as such genders often get lost or muddled along the way.
One thing that has often intrigued me and which I never saw an explanation for is why older languages often have more elaborate and complex grammar than modern languages, and why English has simplified over time more than others. German seems to have hardly simplified at all outside of the forced spelling reform. The simplification of English grammar has been continual and is ongoing today - we're in the middle of losing the distinction between "less" and "fewer" for example, and "whom" is virtually dead. Presumably there's a grammatical limit somewhere, and presumably very old languages were simpler than medieval languages. But Latin is often held up as the archetype of crisp, consistent yet complex grammar and that's thousands of years old.