Having the browser be able to track and tell me that "Though we aren't sure this is actually google.com, we do know that the exact same cert has been used the last 50 times you visited this website" is something I'd consider to be useful. (Actually, telling me if it changes would be the useful bit).
That would be at least be useful for self-signed certs (though those aren't really needed in light of Let's Encrypt...)