This could be accomplished with a legal mechanism that allows the defendant to counter-sue the plaintiff as a patent troll. In this scope they could bring in previous litigative history of the plaintiff (or any related entities, piercing the corporate veil). If the judge rules in the defendant's favor, then all of the plaintiff's patents (and any related entities) are immediately released into public domain.
This would go a long way to preventing patent trolling. And not just because of the direct incentives. Any entity engaging in a patent lawsuit would put their entire portfolio at risk. The larger the portfolio, the higher the risk. The individual or small company with one or a few patents has very little risk. Whereas the large company that hoards thousands and thousands of patents would bear enormous risk on any single lawsuit.