$ apt-cache rdepends libssh-4
    libssh-4
    Reverse Depends:
      libpam-x2go
      yafc
      x2goclient
      tmate
      remmina-plugin-nx
      remmina
      openvas-nasl
      libopenvas9
      libssh-dev
      cockpit-bridge
      kio-extras
      hydra

This is all of the packages that depend on libssh in Debian Testing, doesn't seem to be very problematic, I've only heard of one thing on this list, and that doesn't use the server code anyways.