During the Y2K panic Sun Microsystems (IIRC) announced that they would pay a bounty of ~$1,000 per Y2K bug that anyone found in their software. As you noted, there was very little automated testing at the time so these problems were really hard to discover.
James Whittaker (a college professor at the time) worked with his students to create a program that would parse Sun's binaries and discover many types of Y2K bugs. They wrote the code, hit run, and waited.
And waited. And waited.
One week later the code printed out it's findings: It found tens of thousands of bugs.
James Whittaker went to Sun Microsystems with his lawyer. They saw the results and then brought in their own lawyers. Eventually there was some settlement.
One of James' students bought a car with his share.