Do you think that humans universally or even have historically interpreted physicalism as a simpler theory than a non-materialist one? I don't think so
I think you speak from an already materialist perspective, in that you accept the physical world as something that really exists out there, and that your own subjective being is less tangible or somehow illusory compared to that. That is the opposite way round from how Buddhists felt it obvious to consider it, which was to always frame everything from the first person, human experience (the five skandhas, the eight consciousnesses, etc.)
Personally, I think it's simpler and more obvious to have that first person explanation from the point of view of where I actually seem to be, rather than an explanation that everything I see is real but somehow I am not. Why would I see my immediate experience of reality as less real than the objects that I perceive to be external to me?