Probably because there are international agreements with the minimum fields required on travel documents and "Sex" (Yes, Sex - not Gender) is listed as a required field (though actual values permitted are not specified from what I could see, so 'X' appears to be perfectly valid to add if a country wishes to do so on their documents).
https://www.icao.int/publications/Documents/9303_p3_cons_en....