You don't prove it, the other party proves it to the judge by their inept case.

I think limits should be placed even on this, as poorer people are more likely to have a bad case as a plaintiff as they can only afford to personally represent themselves.

For those representing themselves it should start coming up a bit before they'd be declared a vexatious litigant. For those represented by an attorney, it should probably be the attorney paying the fee for wasting the other party's time and money.