That is not a crazy idea at all. I'd feel a lot safer if I could run some detection on my personal Downloads folder, and on my apt feeds.
The problem is, Linux antivirus does not really exist.
Granted, there is some software that will scan your files for known signatures - but those signatures are for Win/Mac malware, which solves a completely different problem.