Is physicalism the problem? My understanding is that physicalism is still in good standing – at least when it comes to the conscious experience of qualia. (The concept-of-self encoding obviously doesn't require a physical substrate.)
Anyway, trying to understand consciousness by applying physics is just as silly as trying to understand why your complex fluid dynamics simulation is giving an incorrect result by peering at the CPU's transistors with an electron microscope: it's the wrong level of abstraction.
One thing that seems to be making our task a lot more difficult is that evolution doesn't even try to design the biology of systems with clean abstraction boundaries that we can understand. There are so many strange things going on in our bodies that we can barely describe them, never mind understand them. One example is what happens when a person consumes Ayahuasca [1].
I think Chalmers helped clear the air a bit here, though I don't think jumping straight to panpsychism is particularly helpful to the discourse, either.