This matches my experience almost exactly. I wasn't sure that I needed a smartphone, so I got a low-end Android handset on Verizon (Droid Eris), and it's been an absolute piece of garbage, even with custom ROM (currently running a Gingerbread ROM based off CM7). The higher-end Android models I've played with have been better, but even my girlfriend's HTC Desire has boatloads of issues, and some of them are the same as the issues on my Eris (scroll lag, menu lag, poor input responsiveness across the board).
Google really needs to focus on Android's user experience if they want to be a serious competitor to Apple across all market segments[1]. That said, I doubt that Google is going to place a strong focus on fixing Android's UX issues, because it seems to me that Android is yet another platform to get more ads in front of more eyeballs, and if shoveling OS updates and cheap handsets out is the way to do it, then that's what they'll do.
[1]: While it's true that Android has the #1 market share right now, I doubt that it's because of Android's UX attracting the masses--I have a feeling that it's due to Android's low price and availability at said low prices in emerging markets with many fresh-faced consumers (China, India come to mind). I strongly suspect Apple still kicks Google's ass in the all-important sector of consumers with money to spend on phones and apps.