Your device is slow because apps are constantly triggering low-memory conditions. Memory was very tight on the 3G, even before iOS 4. This is precisely why multi-tasking isn't enabled.

Safari alone is consuming most of your device's memory, by a very large margin. It stays resident in memory when you return to the home screen, and it's one of the last apps to be killed in low-memory conditions. The problem is that much worse if you do a lot of browsing on your iPhone or iPod; Safari just grows and grows. This is why the problem appears to be fixed when you reboot your device.

Here's the tip: You can quit Safari by closing all of its tabs and then returning to the home screen. You should do this periodically; I did this whenever I was finished with a browsing session. With Safari out of the way the rest of the software on your device should run at normal speed without triggering any low-memory conditions.

My primary devices now have 256 MB of RAM instead of 128 MB, and I no longer have to worry about Safari's memory consumption.