SPDY performance on mobile networks
googledevelopers.blogspot.in
googledevelopers.blogspot.in
Most importantly, they did in fact not measure anything on mobile networks or even simulations of mobile networks. Instead it was a just a crudely throttled wired connection, which is a poor approximation for how a 3G network behaves. The right way to deal with changing network conditions is to actually do lots of measurements. This is particularly problematic since one effect of parallel HTTP connections is to isolate the congestion control effect of a random network event to just one of the 6 connections to a server.
Second, they are not fetching the actual web pages, but getting all the copies from a local cache. This totally removes the point of testing against multiple web sites. In the real world there's huge variability in things like web servers and TCP stacks / options.
Also, if I understand the replay tool correctly, it's not accounting for the network latency on the inet side of the test setup at all. I'm not completely sure whether this is benefiting SPDY or HTTP though (on one hand SPDY suffers from the latency at the start since it can't accelerate the single flow as quickly during slow start, while HTTP suffers from the latency). But it certainly removes the very common effect of there being a one or two outlier resources being loaded from slow servers.
Removing DNS overhead completely is a bit strange too, and would have the effect of magnifying any relative speedups / slowdowns.
Disagree. Different websites have different structures of content. eg some sites may rely heavily on external resources, while another has everything inlined, etc.
I'm the guy who did this study at Google. I agree with your criticisms that the measurement setup does not reflect real network conditions in all cases. However, we have done a tremendous amount of measurement on real mobile networks as well, and settled on this approach after a lot of iterating and tuning to come up with what (we believe to be) a fairly realistic setup that - most importantly - achieves repeatable results. This was a deliberate tradeoff.
We have measured mobile networks all over the world and observe that the network variation across space and time is incredibly high. It's pretty much meaningless to talk about "real mobile network conditions", since the conditions you measure in an office in Seattle are so vastly different than what you get on a train in Finland or a highrise in New York. The simple throttling we did does, in fact, accurately reflect HTTP and SPDY load times for those specific network conditions, which were meant to be representative of major carriers in the US.
You're right that we did not account for network latency to the origin server, but I would not expect that to skew the results in favor of either protocol, since the last mile is the slow link in this case.
The DNS overhead was an artifact of the test setup, but HTTP and SPDY should have the same number of required DNS lookups in both cases.
Feel free to drop me a line (my contact info is on the original article)!