The universal nature of mathematical truth isn’t dependent on agreement. Instead it’s only universally true when you include all those seemingly unspoken “agreements.”
It is by definition. If you disagree, go write a mathematical proof that 1 = 1. There are many in the world who would love to see it.
But I'm willing to believe I'm simply naive here. You say there is some set of axioms which prove that 1 = 1. Are those axioms not agreements? Since you seem to be familiar, what are those axioms?
There are sets of axioms where 1 = 1 is false, different sets where it’s undefined, and finally sets of axioms where 1 = 1 is true.
However, for a given set of axioms there is no choice and nothing to agree upon. That’s what makes math universally true.