Absent unexpected security issues which are usually patched very quickly as soon as they are discovered by legitimate researchers/white-hats, it is non-trivial to escape one. You are not supposed to be able to escape a VM.

If someone's targetting you with a 0-day exploit that can escape VM sandboxing, having your ssh key hijacked is probably one of your lesser problems. :-/

(If someone has a VM-busting 0-day, they're probably using it in a targetted fashion. The wider that kind of thing is used, the quicker it will be noticed, and patched, and made useless.)