E2E encryption and perfect forward secrecy doesn't matter if the protocol is leaking copious amounts of metadata about who you're talking to and when.
You can see exactly what Signal is able to provide to a legal request. Time of last connection related to a phone number