Ask HN: How Is Google Analytics So Damn Fast?
It updates in real-time, it certainly can't be using bigquery as I see every query in bigquery takes atleast 2 seconds. So, how is it so realtime?
E.g. big aggregates and time series are likely updated on a lower interval whilst near-time event data is probably stored in fast recall storage so that you can satisfy near-time queries immediately.
Otherwise, their long view queries are always sampled. They control the fields available so they can make sure samples are properly stratified in order to be representative.
IMO, it's just good old fashioned good engineering. It could be done with no special tech whatsoever. E.g. sharded Postgres DBs and network storage would do just fine.
- If you are logged into the Analytics dashboard, then a web-socket has been activated between your browser and GA. For fast lookup, there's probably an in-memory hash of GA IDs, web-socket IDs.
SO:
- User visits your webpage / performs some action worth analyzing.
- GA script gets executed; sends request to GA server.
- GA server acknowledges reception and schedules jobs for:
-- storing the hit for associated GA-ID.
-- sending info of hit to websocket associated with GA-ID.
You could look at pre-calculation, subsetting the data, in-memory databases, dedicated servers, non-blocking requests, etc.