Neither Flash nor Java has the best security track record. Java lately has been embarrassingly badPeople say this sort of thing a lot, and obviously with some justification.
However, it's not as if the browsers themselves have a great track record on security either. Firefox had to yank their last update shortly after making it available because of a severe problem, and the update was itself intended to fix quite a few serious security vulnerabilities. Every month we get around to update day for Windows and there are typically a handful of security fixes for IE pushed out. And so on.
It seems fairly clear by now that no-one actually makes a really secure browser yet, with or without plug-ins. It's just a problem that we haven't yet learned to solve, at least not without sacrificing some other benefit that the teams making mainstream browsers value more and choose to prioritise.
If we need independent security tools to keep browsing safe anyway, is avoiding plug-ins any more than a modest improvement?
(And I write this as someone who did once get hit by an undisclosed and unpatched exploit in Java, resulting in a complete reinstall. It wasn't fun. But I don't suppose it would have been any more fun if it had been a zero-day vulnerability in the browser itself.)