From experience, what you care about in that case is to make sure you implemented the private functions correctly, because it might be some tricky algorithm or whatever. Tests with a lower granularity than the public API make a lot of sense to achieve that goal, it's pretty similar to TDD (though you do not necessarily have the write the tests before the code). On the other hand, in the vast majority of cases, the code, once written, will not be changed. If the public API is changed, then it's likely the internal implementation will be replaced, or changed. In any case, the "implementation tests" will break because the contract has changed. So I delete them, the maintenance cost is higher than the long term value they bring, and non-regression should be ensured by the public API tests anyway.
Sometimes, this is wrong, and you will actually need to extend the code in such a way that keeping the "implementation tests" would have helped. I haven't found a good way there to store the test "just in case". You could for instance have a commit that adds them then a commit that removes them, but this is a lot of noise in the history, and the probability that anyone but you can find the deleted tests if they need them is pretty much 0.