Is there a common repository of patent troll claims? It would be nice to have a list of things to avoid, with workarounds where possible.
Basically I would look at what patents the fledgling startup had filed. Make sure they didn't suck, and that there wasn't lots of prior art.
You can be a start up and get bought for nothing more than a single patent that someone wants, so making sure the patents you have are not going to fall down when a bigger company wants them was important.
Often I would also be checking who beat the company to what ever they were doing, and then check those people's patent portfolio.
We didn't worry much about things like the Fat32 patent, we worried about things like hey, this isn't a real innovation these guys just read this other guys thesis paper and knocked it off. (Like Nick at Summly did)
From there a VC could decide if they were going to double down or cut their losses.
There's an old quip about a former president of the American Trial Lawyers Association (now called the American Association of Justice). When it was said that they practically own congress, he joked that he took offense to the word "practically". Even though that's an old story, said in jest, I don't hold out a lot of hope that a group primarily made up of lawyers will clean up a system that benefits their own profession.
Like linked-in, but not voluntary.
And whenever you were about to deal with someone you'd look them up on the list to see if they were bad...