We always responded that adding any extra "weight" to a page made it slower. We didn't give our own tools any special treatment. It was up to the site owners to decided what costs were worth their corresponding benefit.
Put another way: the fastest page is an empty one, but that's not terribly useful. Optimize for not just for speed but for utility.
(Disclosure: I still work for Google, but haven't worked on Pagespeed - or anything remotely related - in many years).