Ensuring you received an unmodified phone without a nefarious operating system or baseband is probably harder.
Not on iOS.
No, they have reproducible builds.
Everything else you said apply even more to other apps, not sure why you think Signal is more suspicious in that regard. But with reproducible builds and the .apk they have on their website, you can check that the source matches it every time you update, which is not something you can do with most other apps.
Ah, I see on Android they do. Not iOS according to an open github issue since 2015.
I am suspicious of signal because everyone says "use Signal it is secure" and it is popularly known for this. Because of this reputation for security, users will get comfortable and talk about their illicit activity, which makes it an attractive target for power brokers the world over. Every app could have this problem, but Signal is the one everyone keeps saying is secure, so it's the place people will spill their secrets.
Reproducible builds on android look cool, but I wonder if there are other threat models on Android than dishonest builds of the program itself.