That is a reasonable way of looking at this, but the question is which concept belongs on your documents. The corresponding field on my (EU-based) passport says sex not gender - don't know how it is in the US.

Your sex is something that is fixed and can be verified (as noted, with very rare exceptions). By this logic, your gender is your own decision. So I think sex has a better place in ID documents. Also I think having the biological sex on your documents can be useful medically - surely you want emergency responders / doctors to be able to tell your biological sex, for medical reasons.