Thing is, it's very difficult for me to understand. But I realized that if I put on CC in French, I can understand it no problem, almost as well as if the CC was in English. But without the CC, I can't understand it well enough to enjoy the show.
This leads me to believe that my difficulty in understanding spoken vs written French might not be so much grammar and vocab of spoken French, but rather the difficulty of translating sounds to words in real time.
I really should leave the CC off entirely, but I want to enjoy the show, so for now, my compromise is to watch with CC in French. I am almost certainly compromising my language skills this way, though.