Well, have you tried to properly study English, as in, with a teacher and a course book? From your description I understand that you merely tried to casually pick up English.
If so, it's not particularly fair to assess the difficulty of English compared to your own language (that, besides, being your native language, you were taught it's grammar and syntax for years on end at school).
I've studied English for several years and have no trouble speaking or reading it -- despite not having had many opportunities to speak it with native english speakers until much later in my life. Reading a lot of books/magazines/etc helps a lot. Reading for like 1-2 hours per day.
I found out I could pick a specialized subject I was interested in, like, say, electronic music production, and just by reading the magazines (Future Music, Electronic Musician, Keyboard etc) for a couple of years I could go from a total newbie to understanding all the specialized vocabulary used (from simple stuff, like "knob", "fader" and "slider" to "dither", "LFO", "frequency cutoff" etc).
Watching movies and tv series without subtitles also helps tremendously. In Italy this is even worse, because they don't just add subtitles, they usually dub the whole thing with Italian actors (ugh).
(OTOH, ho studiato Italiano per tre anni, in un gruppo picolo, e questo e tutto che lo ricordo, ho dimenticato quasi tutti gli parole e la majorita de la "syntax" e gli tempi -- damn, that got bad quickly)