While this is normally done due to the reasons mentioned, to me that is a significant downside.

Why can't GCS act as a CDN, too?