So then what happens is that Google does a fetch from your server, and optimizes the response using mod_pagespeed techniques (including throwing your static assets onto their CDN and other funky stuff..).
Basically, all your CSS/JS/page-content/images get fetched through google's CDNs but this functionality relies on one critical part of the framework functioning really well - Google's connection to your origin server.
This is where it's at: https://developers.google.com/speed/pagespeed/service
Also, yeah my site is for the world at large and that's why this is an issue. I can definitely see how it's useful for personal websites and small projects.