Safari alone is consuming most of your device's memory, by a very large margin. It stays resident in memory when you return to the home screen, and it's one of the last apps to be killed in low-memory conditions. The problem is that much worse if you do a lot of browsing on your iPhone or iPod; Safari just grows and grows. This is why the problem appears to be fixed when you reboot your device.
Here's the tip: You can quit Safari by closing all of its tabs and then returning to the home screen. You should do this periodically; I did this whenever I was finished with a browsing session. With Safari out of the way the rest of the software on your device should run at normal speed without triggering any low-memory conditions.
My primary devices now have 256 MB of RAM instead of 128 MB, and I no longer have to worry about Safari's memory consumption.
It's much easier and quicker to quit Safari by closing its tabs.
I had some shenanigans with it being an older backup and so had to re-transfer my email accounts from my compy to the phone, but that was just a couple checkboxes in iTunes and setting up MobileMe.
The problem is that it wasn't this slow with 3.x, yet I've really not gained any extra features, so it's basically a downgrade.
The new mailboxes make it worth it for me.
Jailbroken? Don't enable wallpapers and multitasking. You'll find out why it isn't supported officially by Apple. Absolute shit performance if you do.
Oh well back to 3.1.2 for me.
Once i freed some space the odd app crashes seemed to stop.
Go to Settings>General>Reset and select Reset Network Settings.
Do that and a quick reboot and you should be all set.