>Trans man: Well that's what being a man means. It's transphobic to think anything else.
I don't know what trans men you've met, but this is laughably far from my near-universal experience of hearing them say things like "oh god what if the way I assert masculinity makes someone feel bad or invalidates someone else's feelings or experiences." I have no doubt people have said things akin to that (being trans is by no means an inoculation against horrifically bad takes) but basically every trans person I know explicitly has a model of gender that doesn't invalidate your experience of "I don't experience gender like that, but 'man' works well for me, not least because of my physical body."
People may point out that it could be good to pull on that thread and consider the possibility of being agender or otherwise non-binary, but no one I know would call you transphobic for critically engaging with your gender and coming to the conclusion "nope, still don't get anything new from considering this, 'man' it is". Quite the opposite in fact, as even attempting to do so should be a decent signal for empathy with trans people.
At the end of the day, the label is for you however that manifests; whether it's having a mental model of gender where you finally have a place instead of always being pushed to the side, or having a magic phrase that indicates to someone the broad strokes of how you'd prefer to be addressed, the label is only important insofar as it helps you.
>If we should [talk] about "people who menstruate" rather than "women"
Just to clarify, "people who menstruate" isn't woke code for cis women to make trans women more comfortable. It's explicitly inclusive of trans men since many do menstruate and strongly prefer not to be labeled women as a result of that.
This is what makes Rowling's take on "people who menstruate" as well as the strong chorus of "trans women are women" and much weaker echo of "trans men are men" in response to it that much more tone-deaf. Trans men were the people whose experiences Rowling aimed to invalidate, but as usual trans women became the public face of the issue.
>Taking the "feeling" of being a man/woman as definitive is exclusive because not everyone who has other gendered traits has such feelings just as taking physiology as definitive is exclusive because not everyone who has other gendered traits has the same gendered physiology.
For argument's sake, what if your subjective experience of being a man were just so ingrained in you that you never consciously engaged with it? That could manifest the same way ("i don't 'feel like a man', that isn't how being a man works for me"), but now would you have a place in the "feelings" model you've established as exclusionary.
It seems to me that the fundamental issue you describe comes down to the coexistence of "I'm a man because that's just what I am" and "I'm a man because I feel like a man", and the only conflict inherent to that comes when the former group feels pushed out by the latter. I personally fail to see any way that someone embracing masculinity, especially if it's something that's long been denied to them, invalidates the experience of a man who's comfortable enough being labeled a man even without subjective experience of his gender.