Yeah this has always been a bit of a pain. Since LXD recently launched a web interface that includes graphical console support (plus VM support some time before that, it's not just a container host anymore, it can do both), I have been using that for remote VMs now. But it still has quite a few rough edges, but nicer than libvirt at least.