Signal actually doesn't know all your contacts - you can check the source code to confirm that it doesn't know about any contacts that you don't message using Signal, for example.
Signal also doesn't store most of the metadata that it could, so it really knows incredibly little about its users. It knows (for example) the last date that it was able to talk to a particular device, but they don't store historical data for that, so if you received a message on Signal today, they don't (anymore) know that they sent you a message yesterday, or last month.
Of course, that second part all runs server-side, so you do have to trust Signal when they describe their internal architecture. But to be frank, who do you trust more with that metadata: Moxie Marlinspike, or the government that is essentially the "sixth eye" in the Five Eyes alliance[0]?
[0] https://www.defensenews.com/global/europe/2018/02/05/french-...