Disclaimer; Signal is probably my favorite app in in the world. As someone that doesn't use any Facebook owned services, there wouldn't be any other way for me to chat with my friends and family.
But I'd really like to understand what you mean and if I am falling for a scam. How does disclosing my phone number make it insecure and not end-to-end encrypted?
I've put Whatsapp for broadly the same reasons. So not the Op, but my 2 cents:
I cannot listen to any discussion of "privacy" or "encryption" when the software doesn't let me create multiple anonymous accounts on computing device of my choice, and instead it insists on tying itself to my most personal device and ID at the very beginning.
I don't know if Signal does it, but Whatsapp additionally explicitly requires access to your phone contact list. At which point, the end-to-end encryption feels like sarcasm. What's left? You have my identity and you have my social graph. You can tie me to anything you want six ways to Sunday. You've asked for my most treasured things in the first 30 seconds of installation. Everything else feels like ridiculous security theater that makes my life explicitly worse (especially the lack of seamless multi-device support) for positively zero benefit for myself or any of my friends & family.
I understand broad first-principles discussion, but for me personally:
I DON'T care if my discussion with my mother in law is encrypted.
I DO care if everybody I want to chat with gets my phone number and social graph.
Communication platform that enforces "KYC"-like disclosure to the platform itself is misleading their users about security.