The crucial fix in ICS is that it finally provides hardware accelerated rendering of the UI. This is going to go a long way towards finally silencing the complaints about Android being slow and unresponsive.
Honestly I see virtually no difference between 2.3 and 4.0 on the Galaxy S II when it comes to scrolling, etc. Indeed, if you turn on the development option that blinks the screen red when the main thread is blocked, it occurs all the time in the basic operating services and in most apps.
Which correlates with the argument made by Hackman (? Hackwell?) regarding GPU offloading not being the actual issue: Instead it's the OS still not, essentially, giving enough respect to the UI. This still isn't a solved problem, beyond simple brute-forcing it (e.g. a HTC One X).