If you're up for it, have a lawyer send them a nastygram/C&D and post on HN about what happens. Many people would probably be interested.

I'm not a lawyer, but my understanding of libel is that at least in the US you need an element of intent to be guilty of libel. Simply being wrong about something isn't libel unless you can prove they know it was wrong or should have known it was wrong. Unless you can show Google knows your site isn't malicious and is labeling it that way anyways, it seems more likely it would be an open and shut case in their favor. I'm also fairly confident Google's lawyers have put a fair amount of thought into this and are already prepared to demonstrate how their labeling is done in good faith, etc.

I didn't do any downvoting, but I suspect it could be a reaction to the fact that treating this as libel would make operating any security company or service effectively impossible by requiring zero false positives. That seems like an obviously unworkable standard even if getting incorrectly flagged as malicious admittedly does suck.