I'm curious, implementation-wise, what would need to be done to do this in a more unstructured way. Rather than having a regular polyhedron and a corresponding texture per polyhedron vertex, it would be interesting to use automatic measures of viewpoint entropy to select a smaller number of views that still adequately describe the object to some level of quality. [0] Rather than it being an octahedron, it would be some other irregular shape of points projected onto a unit sphere, from which a 3D convex hull is computed.
The one issue I can think of is that you'd need a way to quickly determine which triangle of that hull the camera is inside (to get the three surrounding views). Is there a way to do that all in shaders without costly raycasts on the CPU side?
[0] "Automatic View Selection Using Viewpoint Entropy and its Application to Image-Based Modelling", http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.96....