The worst case of this for me was a completely static site which, sans images) loaded in under 100ms on my local machine. I inlined styles, all images with width and height so there is no reflow when the image arrives, no chaining of css resources, deferred execution of a single javascript bundle, gzip, caching, the works. Admittedly it was a simple page, but hey, if I can't do it right there, where then?
Anyway, it all went to st as soon as another guy was tasked with adding share buttons (which I have never once in my life used and am not sure anybody else has ever used).
I won't optimize any pages over which I don't have complete control. Maybe if a project has a CI/CD setup that will catch any performance regressions, but other than that, too much effort, thankless anyway, and on any project with multiple fronted devs, the code is a commons and the typical tragedy is only a matter of time.