I had the same issue until I upgraded to 8 GB. I'd recommend to anybody to go for 8GB directly. For me (or any other heavy user), 4GB is just not enough to run MacOS X (the 'kernel_task' alone takes >500MB).
It made a huge difference. No more constant thrashing of the HDD as it tries to manage a large swap file.