Book 1 is in a class of its own. Books 2 and 3 are a little slower than 1. Book 4 is very slower as you say.

Book 4 brings out the most different reactions: some people even prefer it to 1, some just find it too slow. If you don't like the beginning, I don't think you'll like the rest as it doesn't change its pace. Maybe the problem is reading it in quick succession after 1, 2 and 3. The first time I read it it was after a break from the other 3 and it turned out I now love it (though I still prefer 1). If you're finding it frustrating a third of the way through, I would recommend dropping it now. Then if you feel like it, maybe restart after some time, maybe then it would work, and maybe it wouldn't.

Then books 5 and 6 pick up the speed once more, I find them very fast paced.