I think you've hit the nail on the head. Twitter's UX is painful, but that's really only the half of it. The real issue is that Twitter, regardless of how easy it can be made to use, is inherently useful to a very small proportion of people.
I call it an "extrovert" product. I'm sure I'll catch some flak from bringing up the introvert/extrovert axis on HN (as if we don't get that topic often enough here!). But hear me out. Twitter is for people who like conferences. Twitter is for people who like stages. Twitter is for people who want to have conversations with each other, and also want the world to overhear those conversations. To me, at the other end of the spectrum, that behavior strikes me as weird. It seems painfully and willfully fake and artificial. It seems like so much peacocking, forced smiling, and posturing. It seems like "networking," in the most brutally painful sense of that word. I don't get it. I mean, I grok it, but it holds no interest for me.
As an occasional journalist, I'm often told I "have" to use Twitter. It's "part of the job" to be a "public figure" these days. I'm not interested in being a public figure. I don't begrudge people who are. If anything, I envy and admire their ability to navigate the cognitive dissonance that the Twitter private-in-public dynamic forces. But I just can't do it. It drains me. I find using Twitter to be like doing homework, or eating my vegetables. It's not fun. It's a chore. When a consumer tech product feels like a chore, I don't bother; life's too short.
Twitter is in an unfortunate situation, in that the company was forced to scale massively and go public at around the same time its true user base and use cases were emerging from the noise. Now it's in the unenviable position of having to make billions of dollars in revenue, and of having to compete with the likes of Facebook for a putative billion-something number of users (which it'll never get), when at its essence, Twitter is a restricted product. A substantial portion of the human population will never get any personal utility from using Twitter. Twitter isn't a bad product; it's just not everyone's cup of tea. I don't think there's anything Twitter can do to make me want to use it, and I don't think I'm alone in that sentiment.
It's not you, Twitter. It's me.