I had this problem with my macbook air. It wasn't there when I first got it, but later it did. I think it might have been an update to the OS that caused it.

Anyway, I upped the minimum fan speed to where I still couldn't hear it but louder than before, and that prevented the computer from getting too hot and the fan turning loud. Since then, its been fine, except when running intensive applications (say, Starcraft or Flash video), but that's always temporary and I have headphones on for those tasks anyway.