This is a great idea.
What is particularly galling to me is how media companies trying to protect their content rights make completely meritless legal claims such as mass sending false DMCA requests to video sites saying that they own some work that is completely false, with no punishment for spamming out factually incorrect claims and harming people. That whole messed up system has been commented on many times in other threads.
For actual legal filings, the only issue is that how a "case without merit" is defined is probably very subjective, because legislation can be very complex and lawyers can make a legal argument backed up by some statute for just about anything. There needs to be an extremely high standard for what constitutes a meritless case that should be punished in this way, because otherwise it also makes seeking restitution for wrongs risky, and that's not a good outcome either.
Similarly good ideas in this vein IMO include: simplifying the language of laws, making the actual text of laws more easily accessible, and sunsetting/time-limits on old legislation so that old ideas that no longer make sense aren't kept permanent due solely to bureaucratic inertia.