It's very much a case-by-case thing. On my quad-core Phenom 9650, with 8GB of RAM, I tried building FFMPEG using -j1 through -j10. Build times dropped until I got to -j9, which meant that, for my system, the fastest build occured when I ran -jN where N was 2x my CPU cores. This is the only place I documented it:
http://multimedia.cx/eggs/ffmpeg-and-multiple-build-threads/...