Yes it starts with the senses, but at a certain point the mind abstracts immaterial notions from physical reality, such as justice, truth, point, line, and so on. So the mind must be immaterial.
If there is no coherent notion of physical or material, there is no notion of immaterial either. Chomsky talks about this at length in "The Machine, The Ghost and the Limits of Understanding" (previously discussed on HN here [1]).