If you refuse, then you risk jail (what if it was real?) versus the benefit of keeping the money.
If you get a scam IRS call, then refusing is correct. But the risk-tolerant person considers the alternative: the potential for benefits of compliance over the risk of compliance; they might think "Could be a scam... but I'd rather not go to jail!"
If you get a real IRS letter, then complying is correct. But the risk-tolerant person considers refusing and benefit of keeping their money; they might think "Could be a scam... I'm not paying!"
Anyway, risk tolerance isn’t the right lens to look at this. It’s about knowing whether that’s the way the government would actually issue demands and take payments. That requires some savvy and insight about the world. Some people have it and some people don’t. And relatedly, it’s about being skeptical vs. trusting in general.