Wouldn't other user that sees the other person's profile picture also drum up the cache? This wouldn't work for someone in a large server.
Example you used the normal Signal app without patch and sending me a message, and I have the patched version.
Just to remove certificate pinning, to be able to see the API traffic because of encryption.