Some of those SFP cards can use quite a bit of energy on their own. They're not designed with energy efficiency in mind.

I've got an Optiplex with an i5 and two hard drives and that idles around 12W when idling.