ccgreg

It's as if Common Crawl is attempting to be a sample of the web.

> The CC CDX endpoint, index.commoncrawl.org, historically has been easily overwhelmed and unreliable

Use our Parquet index.

It's also worth noting that archive.org downloads all of our crawl data and adds it to the IA Wayback Machine.

How does that work for non-profit web crawls like the Common Crawl Foundation?

Common Crawl is text-only.

Are you talking about the 8087 transcendental instructions? Which are well known and well-written software doesn't have a problem with it.

Huh. A quick google says that's a feature bit and there's also an OS call to turn it on.

There are? We aren't talking about feature bits, and the only known software that checks the vendor are Intel's math libraries.

Another aspect of the x86_64 architectural swamp is that Intel deliberately makes their optimized math libraries fail if run on a chip that is not 'INTEL INSIDE'. That block totally ignores cpu feature bits.

I've always wondered if virtualization software vendors made everything claim to be INTEL INSIDE, just to avoid this problem.

How is this different from x86_64 and aarch64? Or even the various Alpha and Mips64 chips.

David recently announced he was leaving MLCommons, so maybe we'll get him back as an industry analyst.

Common Crawl has very little content from Reddit and 4chan -- both blocked us years ago.

Common Crawl is about 1 petabyte per year, compressed. Uncompressed is 4X.

Both IA and CCF are private foundations and accept donations.

> It's a shame that CCBot is caught in the cross fire, but that's life.

We're used to it. Sadly.

The author blocked CCBot even though CCBot isn't part of the high traffic problem -- apparently he trusted Cloudflare labeling us as an "AI Bot".

That’s already a big business!

Thanks for the heads up -- this isn't popular yet, and it requires some work to avoid polluting things like the Internet Archive Wayback Machine.

The post says:

> Because Common Crawl stores only the first 1 MB of each PDF

That limit became 5 MB in March 2025.

Appreciate you double-checking.

That isn't true. You're welcome to peruse our index to prove or disprove your claim.

It's kind of interesting that you contradict much of what the article concludes, even though the article gives a lot of examples. Maybe your prediction will be true.

We do. First off we have a public parquet-format index of all of the urls we crawl every month. And then that also lives in a HDFS table that determines when we want to recrawl a page we've crawled before.

Appreciate your kind words! Many people have worked at Common Crawl over the years, and it's been a labor of love fueled by positive comments like yours and the large list of PhD theses helped by our public web dataset.

If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from the AWS Open Dataset Sponsorship Program. And there's no charge for downloading it.

We aren't sure if that really made a significant difference in Common Crawl's data quality. It does hurt our dataset from a humanities point of view, alas.

Common Crawl's archive has metadata that says when each record (html file) was crawled.

Common Crawl's dataset was downloaded in full 100 times in 2025.

We agree that it would be great if it was even more widely used.

A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.

Good timing, I'm about to release that dataset.

Common Crawl is working hard to improve diversity in our crawl.