Only partially. You're right in that some of the binary is cultural, but you're saying, what, "people should just identify as their physical sex, while rejecting the social/cultural norms"? Plenty of people do that. But, sex itself isn't a binary. What makes a person "male" or "female"? "Chromosomes" is the most common statement, and yet, there exist people with XXY chromosomes. There are people with damaged chromosomes, chromosomes that didn't express the way you'd expect, etc. While it can be said accurately that "the majority of males have an X and a Y chromosome" and "a majority of females have no Y chromosome", those generalizations aren't absolute. We don't actually have a biological indicator of sex that is accurate 100% of the time. So trying to define a sexual binary doesn't really make sense either.

So what a person is saying when they say non-binary can vary, and is also quite personal. Which is another reason to use the pronouns they provide (so you can respectfully refer to them using the categorization they provide), and to recognize that what is going on with their genitals, their chromosomes, etc, is not relevant to you.