Eventually I change back and tell the person that they're working for a fraudulent business and should worry about whether they're going to get paid, and should strongly consider getting a new job.
Give the full number and an expired date and go "oh.. this is my old card... I'm sorry... give me a moment, I need to look for my new one1'
I also thought that might be the case but have performed the comparison between a number which never answers them and a number which always answers them. Several years now of experience suggests otherwise: they get the same number of spam calls.
> that they've got a recording of a particular person saying "yes"
Again, this sounds highly speculative. Do you have any citation for it? It isn't like any bank or service is comparing your voice. And if all they needed was a yes, unfortunately, there are too many ways of getting a yes from someone.
Obviously when talking to any of these scammers my 'identity' is entirely fictional.
When they used to be more responsive I had a lot of fun trying to convince them that my routing number was "1" and that my account number was "1"... Or that my name was some absurd 40 character compound name like "Jim Klenersmithvelazquezouishawexlereconomou". A friend once kept a caller from "windows" on the line for an enormous amount of time while they "restored" their system via a recovery disk after it "crashed" several times attempting to "install" their malware.