Apart from the complexities added by having to build this in the first place, this works, but requires a desktop environment, which is not true of ssh.

You could make the argument that this could be an off-the-shelf product you simply install, but then the default port for it because as big of a target as port 22 as soon as it becomes commonplace enough, except you don't have 20+ years of open-source security research in it, and now you're relying on AWS not being now in your region to connect to your servers.