If the value of the work is greater than the fee, then it'd be in the interests of the rights holder to pay the fee and maintain the copyright, for example to be able to license it out to others. If it's not, then the copyright would expire and the work would enter the public domain.
A high enough interest rate for the compounding would ensure that everything is eventually in the public domain. For works that are repeatedly extended, say Mickey Mouse, the public will receive the fees collected for each of the extensions until it enters the public domain.
This also has the interesting effect of incentivizing the creation of new works as there wouldn't be any fees associated with them for the initial term.
There's obviously a lot more details to handle such as having a central registry, inflation adjustments, differentiating between new works and derived works, and how to handle works created prior to the introduction of a system like this. Still I think it's an interesting proposal.
[1]: I don't remember where exactly but probably here on HN.