But overall, I think you overestimate how much time you spend loading the website and how much time it's just sitting there, mostly idle.
And in the end, as long as it's fast enough that users don't stop using the site/webapp/program/whatever, then it's fine, imho. When it becomes too slow, the developers will be asked to improve performance. Because in the end, economics is the driver, not performance.