I understand. It sounds awful. The use-case here being the built-in browser requires host permissions to all url's for navigation purposes. Removing it causing access issues when navigating to a another page through a link within the webpage.
Not sure what permissions are needed to get the current url from the plugin though.
But if it's really more about a good way to browse the way back machine I think a web-service/site would make more sense. Installing a browser extension is quite a hurdle for something you don't need that often.
Too bad bookmarklets are not really a thing anymore, that would work well.