Try my MacBook Pro then, I play movies all the time on it with no problem of frame-dropping or swapping. And it's a 2007 model.
Common problems can be: Spotlight doing indexing at the time, Flash fucking around, too little available hard drive space (less than 5GB), some rogue app, etc.
MBP 2007, 2GB RAM. I have now open: Chrome with 7 tabs, Sublime Text, Terminal (2 tabs, one SSH), Mail, iA Writer, Adium, iTunes, TunnelBlick VPN, Photoshop, Transmit, Dropbox, Alfred, Little Snitch and VLC and the movie plays just fine. I use either VLC or MPlayerX though, very rarely QuickTime w Perian.
Now, some people open 50 tabs and think that the browser should automagically handle them all, with 20 instances of Flash running in videos and apps, etc. Not so. VMs are also very resource hungry.
That said, the laptop is noticeably slower than my 4GB / i7 iMac, but not to the point it swaps --unless I start my 1GB linux VM (VMWare).
On Snow Leopard, I found what you say to be true. On Lion, either the mysterious process is eluding me, or it is just much slower. I do development and I can tell you that the iPhone Simulator is a frequent culprit, VirtualBox is also rough (but if I ssh in to my virtual machines rather than use the GUI it's fine), also the Time Machine daemon trying to backup causes things to slow down (even when not plugged in to a disk), but even after that performance is downright crap compared to Snow Leopard.
If you have any idea what my specific source could be I'd be thrilled to hear it. Frequently at the top of my Activity Monitor is WindowServer and kernel_task, except when compiling or other fun things. Even when the interface is locking up I can't find anything fun.
Recently, work sprung for a Parallels license for me. Yikes, I pity my past self for putting up with all the VB issues because it was free. Just little things like playing nicely with the app/spaces switch key combos, lack of crashes, and performance make me really regret wasting all those mental cycles on VB. Don't get me wrong, having a free x86 VM is wonderful and the VB developers deserve kudos galore... but I reminded myself that my time and sanity are worth a few bucks here and there.
http://cl.ly/1s3N2r2X1u2J2K010H2i
There's no process running wild from what I can see in the Activity Monitor, but it appears to happen a lot less frequently or not at all when I don't have my external hard drives attached.
Also, unless I am running heavy CPU things (XCode, VirtualBox machines, etc), I usually don't get a lot of CPU time like that.
One of the annoying things about Mac OSX is that a rogue codec can mess up ALL media playing activities on the computer- so when you encounter problems, I've found the most successful strategy is to figure out which codec is messing things up.
Other than that, I also don't run Time Machine at all -- I use Carbon Copy Cloner or Super Duper to make incremental and/or bootable backups every week or so. With iCloud, Dropbox etc for the important day to day documents, I don't see much need for Time Machine anymore.
So those would be two places to look at.
Besides those, you can try running some DTrace scripts for a more detailed look when your system starts to crap out.