No. The old commits still exist, a rebase just makes it so that they are not part of the history of the current branch. If you know the commit hash of a commit containing the offending data, you can still check it out. You can also find such commits in the reflog, or if they are part of the history of any other branch or tag. The data is also in git's internal data structures if you know enough to interpret those directly.
Still, a rebase is a critical part of fixing this. If you rebase every branch that has the offending data in their history, and recreate any tags that have the data, then the offending data would be present only in unreachable objects (which can still be found through the reflog or knowing a commit hash). Eventually, git's garbage collection will clean up such unreachable data. You can force this behaviour earlier by using the `git gc` command. By default git will not remove recent objects (which is why you can reflog your way out of a botched rebase), but you can override this behaviour as well.
Of course, all of the above assumes you are working on your own repository. Given git's decentralized design, you need to clean up and garbage collect on every clone that did a fetch since the offending data was checked it. Worse, each clone also keeps track of remote branches, and considers those to be reachable as well, so a garbage collection will not work correctly until a given clone fetches from all of its remotes after those remotes removed the offending data [0].
Further, there is not good tooling to check that you actually did this correctly, so when you are done, you need to hope you fully purged the data.
The plus side of all of this is that once something is checked in, it is extremely difficult to accidently delete it. The downside is that it is also difficult to deliberately delete it. Also, it is fairly easy to make it difficult enough to find to negate the day to day benifets of it still existing, which does very little to protect against a motivated attacker.
[0] Fourtuantly, you do not need all of the remote to have garbage collected, so you could have every make the data unreachable, fetch on all remotes, then garbage collect.