Right, but it’s not uncommon for a repo to be private with sensitive data that is identified and “removed” (using something like bfg or git-filter-branch) before being made public.

Naturally, if it’s a key or something else revocable those extra precautions should be taken regardless of using these tools, but that isn’t an option for some types of data and this implies that users have no systematic recourse.

This is a classic binary security fallacy. It's like saying "there's no point having a lock on your front door because you occasionally leave it open and then anyone could walk in!".

You know you are arguing that it should be impossible to delete things from a website right?