The UI is the connection between the user and the device. When it's laggy and unresponsive, the experience of using the device is always going to be sub-par, no matter how many new features and optimizations the developers have added.
Whenever I've had a chance to test out the latest and greatest Android phone, I've always been blown away by how awful the scrolling performance and animations are, to the point where I really don't understand how people put up with it on a day-to-day basis. My iPhone 3G, at least when it's not randomly hanging due to 4.0, feels more responsive than any Android phone I've tried, and it's running on 3+ year-old hardware (it's got the same processor and memory as the iPhone 2G, IIRC).
Smooth animations, not benchmark results, are equivalent to speed to the average person, and who can blame them?