It’s actually the other way around surprisingly. The big fish tend to have resources to fight the IRS in court; which becomes expensive for the IRS. Hence the IRS goes after the not so rich since they rarely put up a fight.
e.g. https://www.propublica.org/article/irs-sorry-but-its-just-ea...