1) Much (most?) of the time, participants don't get to verify their safety numbers, and in those cases you are at least trusting the server to deliver your messages to the right client. There's a potential vector for man-in-the-middle attack (witting or unwitting) on the server side which shouldn't be dismissed here just because users are "supposed to" verify safety numbers.
2) Their behavior regarding server software might be predictive of their behavior regarding client software in the future. Given network effects, it might not be so easy to leave the Signal ecosystem in the future if your social network is on it, so it's worth knowing right now that it's possible that in the future the client software will also be closed-source.