For the case of ssh versus ssl the detection is trivial: first message for ssl comes from client but for ssh first message comes from server. So server simply waits for a while and assumes that client speaks ssh when it does not receive anything.
One can also match whatever is received in first message from client against known protocols (this approach makes it possible to for example run HTTP and HTTPS on same port as is done by IPP servers, particularly CUPS).