Does that mean you can't do it with some particular view matrix ?
I always wondered if it's possible to have a very low fov near the center, but still account for a large enough part of the screen, so that the fov progressively increase as the image is far from the center of the camera. That way you would have high details near the center of the camera, but more view angle near the edges.
[1] http://en.wikipedia.org/wiki/Affine_transformation#Propertie...
Yep. If you look at the "How?" and "Performance vs Quality" sections, you can see that you need to render the scene six times to get the surrounding environment, so all you would just need to do is make a shader for the projection[0]. (Rendering the scene six times is pretty common in graphics in order to generate light probes for dynamic lighting and global illumination.)
I suspect with a modern GPU implementation (Vulkan) and some other minor optimizations, this could run pretty easily in real-time.