You are right; also I would imagine the tax authority does not even care too much about where the money came from, they just want to make sure you paid enough taxes.
Criminal investigation agencies of course do care.
Usually they do that when they stumble upon them during tax/laundering investigations, but they'll prosecute anything.