Tree Style Tabs itself will need to be redeveloped, but the necessary APIs are more or less available, and there are already similar add-ons that work in Firefox 57:
- Tree Tabs: https://addons.mozilla.org/en-US/firefox/addon/tree-tabs/
- Tab Center Redux: https://addons.mozilla.org/en-us/firefox/addon/tab-center-re...
The most significant missing feature is the ability to hide the native, horizontal tab strip. That's being tracked in https://bugzil.la/1332447.
At the beginning of this thread, the main dev states interest in migrating to WebExtensions, then cites all the bugs that prevent that. Towards the end of the thread, an extension dev (`nt1m`) states that the APIs have been approved, and simply need implementation in Firefox. There's definitely a chance they won't make it into 57, but the rapid-release cycle will help them land in a release not too long after 57.
If TST is essential to your workflow, you may wish to consider using 52 ESR until these APIs are finished in mainline, or contribute implementations of the APIs (time permitting, of course).
Although, I'm on ESR for now, mostly because my company admins cba to patch and enable java web start instead of java plugin for OEBS. Until recently it was stuck on java 1.6. They enabled recent java versions just a few month ago, so I guess I'm stuck with ESR for now.