Ok, I'm actually only 90% wrong! It turns out that there is in fact some information about the BH-NS distinction in the gravitational signal itself, but it is very weak and hasn't been detected yet. It is expected to be detected in the future though.
https://twitter.com/di_goldene_pave/status/13223229779841433...