That's not what I've seen, at least for FileVault 2 under OS X Lion. I'm using a 2010 iMac with a third-party SSD, and a 2011 MacBook Air with Apple's stock SSD. Both computers are blazing fast at disk access, even with full-disk encryption enabled.

If you're confident you're encrypting all your data, then you're still way above average. The nice thing about FDE is that I can "set it and forget it." I don't have to think about which files belong on which partition anymore. It's all safe.