Signal doesn't upload your contact's details anywhere. It hashes your phone number and sends that to a central service that knows which hashed numbers have Signal. Then it periodically asks that service whether hashes of contact numbers are in the list in order to decide whether to suggest Signal for them instead of unencrypted messages.
It turns out that some people genuinely are forgetful enough that if they told their iPhone Bob's number, email address and shoe size in 2016 and then in 2019 their phone finds out that phone number is registered for Signal, they will conclude that the phone must have learned Bob's details from Signal, which in turn stole them from Bob as part of some nefarious plan.
You can't do anything about this, it's like the Spam problem. If you send ten million very, very useful emails that are genuinely valued by every human recipient, hundreds of them will be flagged "spam" because Humans aren't very good at this sort of thing. They press the wrong button or they've been using "mark as spam" because they thought it's "mark as read" or they meant to mark the one below it, or above it.