The gyroscope combined with the accelerometer and the compass is able to accurately find your relative motion in 3D space without any external references (just the initial point) using simple physics. But yes it also drifts but very slowly.
Tracking rotation can be done pretty cheaply & with low latency these days, and I expect the Daydream viewer will probably handle it quite well.
Translation is quite a bit more challenging, and I'm not expecting much in that department for this iteration. The issue can be addressed to some degree with a neck model that maps rotation to translation, but it can't handle real translation, such as leaning or walking around.
This is on par with Oculus/Samsung's current mobile solution, the Gear VR headset.
I'd imagine that this is to be expected as it's a sort of "VR Lite" compared to more fully featured platforms. VR experiences will need to be designed with the limitations of the platform in mind and not rely on moving around in 3d space but rather focusing on head orientation.