I am on facebook, but don't want to use facebook connect. Maybe I'm also a minority I don't know. I sort of like my facebook being separate to everything else.
As with many of the login complaints, this is probably only an issue for a small minority of your target users.