I'm largely in the same boat, but I use nvim instead. I already learned how to manage panes and tabs in vim so once the terminal mode came out I went ahead and gave a try switching.
What I think you're missing with just nvim is the tmux scripting capabilities and of course the session management.
It's pretty easy to script up a tailored set of tmux sessions/windows/panes. I do this on initial login. Other solutions like tmuxinator/tmux-resurrect exist to automate this setup, if you don't want to script it.