I personally use tools that report all the activity that a software produces, most of that is TCP or UDP connections. Among the tools that I use are WireShark, Burp Suite and mitmproxy. I also keep myself updated with techniques used by other researchers in other areas by collaborating on different forums and networking events. I also have several hundred honeypots distributed around the planet powered by OSSEC to collect and analyze bad traffic.
Less than 3 years ago I switched to do vulnerability research on mobile and desktop software and a whole new world opened in front of me. Debugging HTTP connections is one of the most common tasks and there are plenty of tools available out there, I have a small set of tools to do network sniffing and analyzing. Going deeper into the software, I frequently do black-box penetration testing (which basically means, I don't have access to the source code) and so tools like IDA, Hopper Disassembler, Binary Ninja, Cutter are first hand in my toolset.
> How do people there [Google Zero] decide upon a course of research?
• Someone tips you some information about suspicious activity,
• You are using the software as a regular user and notice something weird,
• You are interested to know how a software works and diving into it reveals secrets,
• One of your honeypots and/or network sniffers alerts you about unwanted connections,
• The author of that software requests you to do some penetration testing for an audit,
• Someone found a small problem in the software and you dive into to try to find more,
• And more commonly, you are bored and want to pass the time doing something more boring :D