For me it wasn't so much him as all of the songs IIRC. Once I started just skimming the songs/poetry he became much less annoying (I'm horrible at deciphering meter and it just trips me up). The first words out of my mouth upon seeing the first film was "they left out Bombadil".

Had been looking forward to hearing one of those songs.