No, because of a dense, interconnected web of other social truths (the rest of the arithmetic model), the relative error of this one truth/model is negligible.
However, confusing your model for reality is a fallacy perhaps older than time.
> However, confusing your model for reality is a fallacy perhaps older than time.
I think that the human perception the world is a robust enough model to be equated with reality without issues. If you go down the path of denying perception, you might as well go full solipsism, in which case it doesn't even make sense to discuss reality at all.
The model is "good enough" for the purpose of creating a building, but that doesn't make the act of "creating a building" any less real, nor the underlying rules that govern matter any less true. Our descriptions are not real - the rules themselves (which we may not know exactly) are real.
Therefore, mathematics - the set of rules that corresponds to how reality works - itself exists in reality regardless of social consensus. Society can't change them by making a different consensus.