The polygamy stuff is his books has always been a turn off for me. I don't really have anything against it, but it's that he goes to such lengths to explain and defend it, so many times (and always as some sort of "perfect system"), that it gets tiresome.
My theory is that every Sci Fi author has their own sort of pet belief they like to expose on their novels. So I get it, this is his. But every time it came up on his books I couldn't help and roll my eyes at what I knew would be a long, unnecessary, mostly off topic diatribe about the benefits of polygamy.