You should stop thinking about it and do it. I cancelled mine June of 2010.
I didn't so much "leave" as I just stopped going.
I've taken the same approach. Also no longer using Facebook Connect to sign up for new services - which is one of the only reasons for maintaining an account.