That’s the actual experience. There’s open bugs about it in Cardboard.

Don't worry I'm sure it'll be buttery smooth for real. Graphics on linux are known for that, right? ;-)