It would be reasonable to spot-check driver's license (which they necessarily have for driving said vehicle). Should be easy enough to verify against the handicap permit.
man sometimes you just have to let externalities happen in the name of not screwing over the 99% of people who do it by the book
I don't see how this would be any different.
That being said, IMHO it's better to tolerate a moderate amount of abuse than going over the wall with enforcement.