If a privacy-focused app makes you nervous about privacy they've already lost the war, haven't they?
They have a privacy policy https://whispersystems.org/signal/privacy/
They have explained their reasoning https://whispersystems.org/blog/contact-discovery/
You have access to the source https://github.com/WhisperSystems
You have proof that they don't store your contacts and can't provide them even after a subpoena https://whispersystems.org/bigbrother/
Again, what exactly are you nervous about? What is gained by removing contact discovery?
I don't particularly want to send Signal a list of contacts - I'd rather not leak that information about my social graph. I understand how they want to use that information for discovery, so they can know who already uses the app.. But for my needs, this isn't necessary. I'd prefer to manually ask my friends for their username on the service.
If there were additional options, I might choose to use their service. As it is, it doesn't fit my needs, and that's OK.
I'm sure there's plenty of other people who appreciate it.
Of course it would be nice if we didn't have to trust them that this won't change in the future (whether through a court order or their decision doesn't really matter).