This always comes up when I mention privacy.com. It's always a theoretical possibility. No one's ever shown a case where that actually happened (and it's never happened to me). People also say, "oh, but they'll ruin your credit rating."
You couple the card cancellation with a message to them demanding they cancel your account. Just imagine a credit card collection company agreeing to come after you, when you made a good faith attempt to close the account.