So this isn't a great example because you can indeed see that many requests are redundant and are frequent by visiting a few sites: these facts don't depend on any hardware etc.
Ultimately the source of truth is actual end-user performance. If you’re not measuring that directly, then you’re hoping what you’re doing is a proxy for end-user performance.
Sometimes a suitable proxy is available. Often, it’s not.
The goal isn't (shouldn't be) to just find a bunch of things to optimize but instead to achieve the best balance between complexity and the experience for the vast majority of users given limited engineering resources.
> you have no way of knowing how significant it is in terms of how many sites/user/visits it affects
The way of knowing is getting site visiting statistics
> I'm a big fan of looking at actual performance data
And you'd be looking at exactly that - actual performance data on representative hardware on representative websites, but with more flexibility and more data available given the fact both parts are yours and you can collect anything without any privacy concerns
A "few sites" was a quote from your earlier comment. [1] You're moving the bar. Perhaps you're seeing it isn't quite as easy as you first thought to do a good job with this.