Keys probably aren't the best example. What about PII, accidentally committed "test" data, documents, nudes, whatever..
You may get lucky and remove/hide it fast enough, or think you did...
This is an with Github today, all public repos are being watched by bots reviewing all commits for accidentally-pushed credentials.
The only solution is to not use a public repo.
On GitHub yes someone might be watching, but deletes are still possible.