All these things cause tremendous problems for learning the language because you have to track distinctions in the language that don't exist natively so you never had to think about before. Native speakers learn genders just by hearing them repeatedly, at the same time as learning the words, but English speakers learning German normally need to study dictionaries a lot to learn new words and as such genders often get lost or muddled along the way.
One thing that has often intrigued me and which I never saw an explanation for is why older languages often have more elaborate and complex grammar than modern languages, and why English has simplified over time more than others. German seems to have hardly simplified at all outside of the forced spelling reform. The simplification of English grammar has been continual and is ongoing today - we're in the middle of losing the distinction between "less" and "fewer" for example, and "whom" is virtually dead. Presumably there's a grammatical limit somewhere, and presumably very old languages were simpler than medieval languages. But Latin is often held up as the archetype of crisp, consistent yet complex grammar and that's thousands of years old.