Even if you restart the ssh server it won't prevent people from connecting to the server because SSH forks the clients.