After I refactored one of their pages to go from 1.5 sec initial/8 sec total page load to 0.04 sec/4 sec load times there was no discernible change in rankings or traffic.
Page load as SEO is totally over egged for the most part as it's one of the things developers can easily measure and point a finger at, along with number of requests, CSS file size, etc. while everything actually meaningful for SEO is absolutely black box.
Every time I get handed a "SEO site report" by a client I can immediately tell if the SEO guy is any good or not if the report actually contains anything else other than recommendations to improve page loads, reduce requests and all those practically useless, but very expensive in developer time, actions. Oh, and the obligatory spreadsheet with all the external inbound links domains. Because that's really insightful.