I think it may have something to do with background images and/or transparency.
Even some quite prominent sites suffer from this flaw (making them almost unusable for me):
Now I realized I have seen a naturally occurring test case: Twitter user pages with and without static background images. Everything else is the same, yet the ones with the static background image (CSS => background-attachment:fixed) are much slower.
I'm on Windows XP with Firefox 3.1 Beta 2. I have the same trouble in release Firefox and latest Chrome.
Explorer 7 and latest Safari is fine. Latest Opera is ok, though JayIsGames is killing its silly wait-till-all-small-stuff-is-loaded rendering engine.
BTW I see this problem on many sites. These two were just ones that popped into my mind.
Maybe it's a difference in some algorithm between FF/Chrome and Explorer/Opera/Safari, though Chrome and Safari share at least most of their rendering code. That leads me to believe that it's probably something to do with Flash or something embedded. I don't know what to tell you, though, I've never had any problems, and my computer isn't even that fast.
What are your computers' specs?
What's interesting is that for me page scrolling speed isn't spread over many values for different sites. It's either fast (majority of sites) or very slow (some sites).
That's why I didn't suspect computer speed. It did look more like rendering engine bug / inefficient code path for particular layout/image/CSS combinations.
It seems like some designs trigger heavy computation, which is not noticeable on faster systems, but is very painful on slower machines.