Indeed, it definitely is one hell of an assumption. Just as "matter is fundamental and consciousness is an emergent phenomena" is. Until we find a way to test it in any way (which may not be possible at all) the only thing we're left with is to play with the axioms and get a feel of what the resulting theory can and cannot explain.

However when we accept the 'panpsychist' axiom, we're free to come up with more detailed theories (as Integrated Information Theor) that eg. predict under what conditions physical systems and consciousness coincide in a way we can experience. And indeed they claim they can explain eg. why brain, as opposed to silicon circuits, coincides with it.

To be clear, I'm not saying this theory is correct - I've no idea. I'm just saying that this whole approach is as valid as the physicalist one.