That has always bugged me about Signal. It seems like such an obvious hole--one that other services like Telegram and Wickr don't suffer from.
I think Signal implies that you always know and are comfortable with the person on the other side of the message, but that is not always the case.