I think that things like metasploit and heartleech are an almost purely unalloyed good. In my experience, the "bad guys" already have easy-to-use tools. What publicly available tools do is give defenders access to these techniques, which they can use to demonstrate that problems really are a Big Deal. There is a certain kind of person (who seem to gravitate towards management) that cannot be convinced to take an issue seriously unless they can see the impact with their own eyes. A tool that prints out the private key of their production server is worth a dozen blog posts and security advisories as far as convincing them the danger is real.