I lived in Tokyo for 2 years, studying Japanese full-time, and completed several courses in Finland before that. Speaking came pretty easy and I am quite fluent in normal conversation, but learning kanji has proven to be a real time-sink and a source of frustration, although at the same time I love them for their beauty. Also reading Japanese feels different than reading languages in the Latin alphabet, as the characters have an extra layer of meaning (not a huge difference, but helps to distinguish homonyms + create new kinds of puns / emphasis).
There are roughly 2000 kanji that you should know to be a high-school level reader. The latest test I took (http://www.speedanki.com) shows I know about 800. Reading a newspaper is not possible currently, as I would have to constantly look up kanji, and often the important words are the rarer ones.
But I will learn them. At this point it's an obsession, I'm not even sure why I need to know them, except to prove to myself that I can.