I'd recommend Qutebrowser as a replacement: https://qutebrowser.org/

I switched over from Pentadactyl to Qutebrowser last year and it's way snappier. And with the experimental WebEngine backend, I haven't run into any rendering or performance issues at all. It also has a vertical tab mode, although it isn't possible to have sub-tabs like Tree Style Tabs.