Very little goes through google's servers, actually. "Casting" something from your phone/tablet to the chromecast is just sending it a url (and a timestamp if you were already watching on your phone). Then it streams locally.

So, for example, you can stream from Plex and not touch Google servers at all.