A way to stop frivolous lawsuits of all sorts is to have the loser pay the winner the lesser of (what the loser paid his lawyer) and (what the winner paid his lawyer).
For instance, much of our civil rights enforcement comes about through private civil suits for employment or housing discrimination. We use malpractice lawsuits as a major part of dealing with bad doctors. We use consumers suing over bad products as a significant part of keeping consumer goods safe.
Your suggestion that the loser pays the lesser of what the two sides paid their lawyer is interesting, because it would not discourage lawsuits as much as a general "loser pays winner's costs" system, which is what is usually posed. Effectively, in the common case of a small party suing a big party, your proposal would double the plaintiff's cost if he lost. That would probably be a small enough risk that it wouldn't stop, say, a person who thinks they have been discriminated against in housing from suing--it would just mean he'd go with a less expensive lawyer than he might otherwise have used.
> Transparency: The draft includes strong language requiring patent trolls to reveal the parties that would actually benefit from the litigation (called the real party in interest).
> Joinder: If the plaintiff is a shell-company patent troll, the defendant could require the real party in interest to join the litigation. Even better, a prevailing defendant could collect attorney’s fees from the real party in interest if the patent troll can’t or won’t pay.