I believe that only CNAMEs use the CDN for custom domains (naked domains do not) [0]. So that rules out a chunk but there will be lots of cache misses too. Maybe someone from GH can confirm.
[0] https://help.github.com/articles/about-custom-domains-for-gi...