Some people, myself included, are wary of Facebook and don't have an account there, for various reasons. Privacy erosion, nasty tactics, spam are among them.
Now I'm not saying these issues are present in Facebook Connect, but I don't have the time to find out and I don't want to gamble that everything will be dandy in the future, especially with a company so removed from ethics as Facebook.
When you name a new product using your old and renowned brand (like Facebook Connect using Facebook), both positive and negative karma is inherited from the old brand. This also may explain why some don't like FB Connect.
And, if you use another company's product (like with Quora and FB Connect), you risk inheriting the consequences of its negative karma as well.