It's possible to build a speed test site out of media file fetches from otherwise public assets from third party sites, the quality of the measurement will just be a little noisier. In many ways this would actually be a better test than some well-placed IP address, since there is a chance to randomly sample many different routes from the ISP to real world destinations for practical object sizes (i.e., not just an idealized unidirectional stream with no request roundtrips in between).

For example you could do something with Image.onload event, but even better are the recent JS web performance APIs, they provide a huge amount of detail - including DNS lookup latency, TCP connect time, TTFB etc.

Actually now I want to build this

edit: I'm probably not going to build this, but you should. For large object sources, there are Flickr full size images, HLS URLs for video on demand systems (each HLS segment is generally around 1-3MB), etc.

I'd want a giant list of these to pick from at random, and probably want to bucket the results along (domain name, destination BGP ASN, destination GeoIP) axes as well as whatever the JS performance API offers. You could maybe present each axis as some kind of confidence interval, then geometric mean to produce an overall score presented graphically somehow

You could then collect up all these scores and publish stats bucketed by Internet provider. And that's the point where I realized I don't have the energy to build all this ;)