Quite frequently the difference between what you want to protect against vs real threat landscape is what rules the expert community’s decision.
I feel your pain (I work at company which improves data security of distributed systems in a number of ways, and get into similar disputes all the time), but the fact that you’re mitigating some of the risks really well does not mean that other security properties will not be scrutinized against commonly recognized threat models.
Not too rewarding, yet the whole security engineering is a Sysiphus’ labor fron day 0. Not having enough of in-community pressure (even when sometimes you’re criticized for a wrong cause, which is 50/50 here) would lead to much worse consequences than a few uncomfortable questions asked.