I've been a Prime member for over a decade (although I've recently cancelled my Prime membership), and I've made very few complaints over the years. But one of the many things that has soured me on Amazon is that the last three times I needed to do a return (in every case because the item was damaged), it's been a huge pain in the butt. And once, Amazon even claimed they never received the return at all, and recharged my account for the ($20) item.
I know that others have a very different experience than I, and I don't know what the difference is -- but Prime vs non-Prime doesn't seem to be it.