Looks really cool and well done, though I'm failing to see the value this adds vs using standard user access controls and a regular old terminal multiplexer like tmux or screen.

You can even get the same usage model very easy by doing what a lot of companies do by throwing up a 'jumpbox' that all users log in to. They then log into whichever server they need from that session. You can do both PAM based, and network based ACL's and it's fairly easy to manage through sudo etc.