That should be done a lot more than it's usual today. Part of "mobile first" should be testing mobile speed.
One problem is of course that mobile networks vary greatly in both latency and bandwidth. A great resource on performance considerations for mobile networks is Ilya Grigoriks open book [1].