Pocket lint in the socket also keeps it from seating. Digging that out is like playing Operation. The bits you can break are on the sides, so you have to push straight in to scrape out the lint.

I really should learn to put my phone in my pocket wrong way up so it doesn't catch all that crap...