I haven't spent much time with Joyce past Dubliners, but I've spent a great deal of time with Kerouac, and particularly "On the Road".

I don't think it takes a special person to enjoy, but it does require a particular mindset that any given person may or may not have at the time. It's sort of how "Catcher in the Rye" is brilliant when you're a teenager, and how very, very different it is when you're an adult.

On the Road is not the sort of book that I'd spend much time with now, but I've grown towards, and to appreciate the brevity of Hemingway or Elmore Leonard more now, but if you're not inclined towards Joyce, wait a while. If it never happens, so be it.