I've been working on a project to do that, basically I download all the streetview tiles along a route and stitch them back together. Then I turn that into a 360 video and let YouTube handle the rest.

See here a video speeding through South Africa : https://www.youtube.com/watch?v=EtWDF8-gRGk

or rural New Zealand : https://www.youtube.com/watch?v=M9s6Ahxfjmc

Fun to use with google cardboard. I'm still not sure what the best speed is, I think I may have them going too fast still.

I investigated making this recently, however the (free) Google Street View API currently only returns 640x640 images, not the entire equirectangular image [1] that would be needed to create a photosphere for a VR app. So one can make a cool application like OP's here, but creating a panoramic experience would require multiple API calls and your own fancy stitching algorithm.

[1] http://wiki.panotools.org/Equirectangular_Projection

You can do that with google cardboard!