Not so long ago I side-project fassttt.com and thought the biggest issues are server-side, with bad images and improper caching. Ended up creating a complete minifier/compressor/cdn-enabler.
I since realized the problems are mostly client-side. Cumbersome styling framework and JS rendering engines that are dynamic and VERY hard to optimize.
BTW, if people only care about site-speed score, see https://nitropack.io/ which almost always give you a 100 score.