Leaving the tab open for a half hour turned up about five of those, a google oauth key, and what seemed to be an SSL certificate private key, among others.
The usual format of private keys makes it mechanically trivial to get the corresponding public key back and then you can ask a public monitor like crt.sh to determine if that key is seen in any certificates.
Once you identify one or more certificates, you can ask the issuer to revoke them, offering proof you know the private key as the reason. For Let's Encrypt or any CA offering the ACME protocol you can use the ACME protocol to do this without involving any humans (the Certbot software for example has a "revoke" keyword that can do this).
Other CAs should have at least an email address manned 24/7 to respond to problems, which can explain how they'd prefer you prove you know the key. Or if you're just bored of this and want them to shut up and fix it, you could just email them the private key, whereupon now _they_ know the key for someone else's certificate they issued and that's prohibited by the rules they're working under so it's now their problem.