So the moment something is published on the Internet publicly, there's a chance it will be saved and you will not be able to get it deleted.

That, unfortunately, sounds like the result of publishing something on the Internet. Not GitHubs fault.