If you comply, then you risk losing the money (what if it was a scam?) versus the benefit of avoiding jail.

If you refuse, then you risk jail (what if it was real?) versus the benefit of keeping the money.

If you get a scam IRS call, then refusing is correct. But the risk-tolerant person considers the alternative: the potential for benefits of compliance over the risk of compliance; they might think "Could be a scam... but I'd rather not go to jail!"

If you get a real IRS letter, then complying is correct. But the risk-tolerant person considers refusing and benefit of keeping their money; they might think "Could be a scam... I'm not paying!"