You can check it yourself with reverse engineering tools, e.g. on Android you could look at the APK with jadx and see the decompiled Signal implantation, and how it compares to the open source Java library from Signal that it is derived from.
I've reverse engineered file types. But not actual code.
On the other hand, maybe your binary can be decompiled to source code (e.g. CFR can decompile java classes, I've used that a bit but haven't checked if that's useful for Android apps).
Maybe this was bad advice, this approach is like trying to learn a spoken language by listening to a recording and looking up each word at a time. Probably it'd be better to get a deep understanding of how CPUs work in general, I can recommend the book Computer Organization and Design by Hennessy and Patterson for this. I agree with the other commenter that 15 years is more realistic so maybe just start by getting a 4 year degree in CompSci (and that book will probably be required reading).
All in all with 15yrs of training you can start the task :D
What you say can be true if source is readable and well written and you have somehow a guarantee that source is not modified before publishing the app. Or if you just are inexperienced with auditing the app other than reading the code (I know I am).
If we're interested in whether WhatsApp's developers have implemented the Signal protocol in good faith, because we suspect they have not (we suspect they've put in backdoors at the behest of bad governments), then having the source available (and reproducible builds to use) allows us to check that more easily than analysis on the binary's behavior.
This is because the primary way the duplicitous maintainers would do this is in their own source. It's possible (and for three letter agencies probably common and desirable) to deliberately introduce bugs in dependencies that provide the desired behavior in some end product, but it probably shouldn't be our base case. Our base case should be: you swear your code implements the Signal protocol without backdoors...ok, great, you show us the code.
Does Signal itself even have reproducible builds?
They are fully entitled to do that of course, as it is their system. But that makes it far less useful for hobbyists.