I finally got through it. Had to read Mason & Dixon first, which was perhaps even harder to start off because of the Old English spellings. But after getting past that and realizing it was mostly just a funny story, the rest was pretty easy and enjoyable. Then coming back to GR was much easier.
Whether it was worth the time invested is another question entirely. ("No", I'm afraid, is the answer. There are better books along the same lines that are also challenging but less so. I recommend 100 Years of Solitude, Midnight's Children, or Tin Drum).
Avoid Ulysses. There's such bad, lame "philosophy of math" in there it really turned me off, made it seem like the whole book is just Joyce showing off how smart he thinks he is.