I do something similar. I set up a "pair" user on the system whose login shell is just a script that attaches to the tmux session. The script that starts the tmux session, sets the group permissions on the socket to "pair". This means that even though the "pair" user has their ssh key in pair's authorised_keys file, they can't log in unless there is an ssh session with the appropriately shared socket. Also, they are never logged in as your user (though, sharing the session, they can do whatever you can do -- only you are guaranteed to be able to see it).