To the card company, the difference between a stolen card that was used by thieves and a seller that fraudulently charges a card is whether you still have access to the card. In both cases, the account holder hasn't authorized the payment, therefore the payment processor had no valid order to process the payment. Since the agreement you have with the card issuer generally only covers authorized payments, they have little ground to stand on unless they can prove that either you did in fact authorize those charges (fraud) or acted with gross negligence (sharing your details with a third party in ways not intended, so not in a payment process).
A payment processor isn't (at least where I am) entrusted with any special powers to make judgement calls on whether you deserve your money back based on a potentially existing contract between you and a third party - that's the court's job. Cooperating with them makes it easier, speeds up the process and ensures an ongoing relationship with them, but you don't have to prove that your card was stolen (which is hard to do anyhow) - they have to prove that it was you that authorized the charge. Getting your money back without their cooperation may involve taking them to court though.
There's a similar thing with SEPA direct debit. You can pull money from any account (it's trust-based; you're required to provide documentation showing that you were authorized to do so when asked), but the account holder can pull it back for a certain amount of time (iirc, it's six weeks, at least here). The banks do not act as a judge here, they simply put your money back into your account and inform the other bank that the charge has been reversed who then in turn takes it out of the pulling account. If the other account holder believes the charge back to be unlawful (i.e. fraud, or or a charge back because of insufficient balance), they have to bring legal action, but they can't use the bank as a tool in the process.