I agree that just equating Commission / OTE = FYE / TAM seems pretty arbitrary, I was unclear how this was "optimal".

In my understanding, commissions are used when effort is not observable but outcomes (i.e. closing a sale) are. An additional point in my head is that the commission / base trade-off is a decision that depends on how sensitive your salespeople are to risk.

I think you would really want to set marginal benefit to marginal cost. In this case, the marginal benefit of increased commissions would be incremental increases in effort and marginal costs would be incremental decreases in customer satisfaction or decreases in employee retention due to risk aversion. Of course, these might seem harder to quantify that FYE/TAM, but I think that the fraction of FYE is a little made up to begin with :)