We know what the IRS knows because the majority of people get the same forms as the IRS as far as income goes: W2, 1099, 1098, ... Unless you’re self-employed, the IRS only misses some deductions or foreign income.
The IRS information is at times comically incorrect in fact.
99.999% of the US population is not trading $20 million a year in stocks, and everyone else here is merely talking about the 95% of the population that has straightforward tax returns.