My reading of 5e says the opposite. That 'unless expressly permitted by content owners' you shouldn't be allowing caching of content provided via Google APIs.
Considering that most (all?) fonts on Google Fonts are open (most I checked use SIL or Apache as license), downloading is most likely allowed.
Think of it like an affirmation from an open source project licensed under an MIT or BSD license. Although they might include GPL code from elsewhere, they'll generally state that they're not relicensing someone else's GPL code under the less restrictive license.
Probably the biggest issue for the service will be to make sure that licensing is ensured. (see https://news.ycombinator.com/item?id=8802833)
This may result in a higher latency, but might be more safe in regards to their ToS.
Your temporary caching, to me, seems fine. I read the ToS as saying that permitting your end user to obtain permanent copies might be disallowed ... at the discretion of the font designers. To be absolutely certain, you should check each font's license. Considering the fonts aren't Google's to license, I doubt you'll hear from them.
Interestingly, Google's own Fonts API which I'm using with my service, doesn't provide any information about the underlying license of each font (I previously wanted to include that info in the UI as well).