It's possible to build a speed test site out of media file fetches from otherwise public assets from third party sites, the quality of the measurement will just be a little noisier. In many ways this would actually be a better test than some well-placed IP address, since there is a chance to randomly sample many different routes from the ISP to real world destinations for practical object sizes (i.e., not just an idealized unidirectional stream with no request roundtrips in between).

For example you could do something with Image.onload event, but even better are the recent JS web performance APIs, they provide a huge amount of detail - including DNS lookup latency, TCP connect time, TTFB etc.

Actually now I want to build this

edit: I'm probably not going to build this, but you should. For large object sources, there are Flickr full size images, HLS URLs for video on demand systems (each HLS segment is generally around 1-3MB), etc.

I'd want a giant list of these to pick from at random, and probably want to bucket the results along (domain name, destination BGP ASN, destination GeoIP) axes as well as whatever the JS performance API offers. You could maybe present each axis as some kind of confidence interval, then geometric mean to produce an overall score presented graphically somehow

You could then collect up all these scores and publish stats bucketed by Internet provider. And that's the point where I realized I don't have the energy to build all this ;)

I believe this is the idea behind the https://fast.com speed test.

It just downloads data from Netflix servers, so the results you get are basically guaranteed to be the speeds you would get with Netflix.

I think the bigger problem isn’t so much the traffic type as it is traffic destination. As long as there are only a few hosts or domains providing speed test services (and there always will be: upload speeds are expensive) ISPs will be able to whack-a-mole with their whitelists.

I’d say that this needs some sort of regulation, but as long as ISPs are the gatekeepers, they can cheese $Government all day long too. This word gets overused perhaps, but the closest I can come up with is a decentralized monitoring setup with random speed test hosts (especially hosted @home style). Care would have to be taken to avoid how-are-these-still-legal data caps though.

This is one of the biggest selling points of fast.com -- it's served from the same Netflix domain + servers that also stream video (nflxvideo.net).

If ISPs want to make fast.com fast, then they need to do so for all of Netflix.