Even more distracting than the shadows would be the fixed perspective. Unless you include some form of head tracking, the immersion will never feel real. Unfortunately, it'll limit one person per room.
Classic example of head tracking: http://youtu.be/Jd3-eiid-Uw