It seems that OpenAI is scraping [certificate transparency] logs

https://www.merklemap.com/search?query=ycombinator.com&page=...

Entries are indexed by subdomain instead of by certificate (click an entry to see all certificates for that subdomain).

Also, you can search for any substring (that was quite the journey to implement so it's fast enough across almost 5B entries):

https://www.merklemap.com/search?query=ycombi&page=0

To me, this is evidence that SQL databases with high traffic can be made directly accessible on the public internet

crt.sh seems to be more accessible at certain times of the day. I can remember when it had no such accessibility issues

P.S. In the hopes of making this more than just a sarcastic comment, the question of "How do people bootstrap knowledge?" is kind of interesting. [1]

> To tackle a hard problem, it is often wise to reuse and recombine existing knowledge. Such an ability to bootstrap enables us to grow rich mental concepts despite limited cognitive resources. Here we present a computational model of conceptual bootstrapping. This model uses a dynamic conceptual repertoire that can cache and later reuse elements of earlier insights in principled ways, modelling learning as a series of compositional generalizations. This model predicts systematically different learned concepts when the same evidence is processed in different orders, without any extra assumptions about previous beliefs or background knowledge. Across four behavioural experiments (total n = 570), we demonstrate strong curriculum-order and conceptual garden-pathing effects that closely resemble our model predictions and differ from those of alternative accounts. Taken together, this work offers a computational account of how past experiences shape future conceptual discoveries and showcases the importance of curriculum design in human inductive concept inferences.

[1]: https://www.nature.com/articles/s41562-023-01719-1

Certificate transparency logs are intended to be consumed by others. That is indeed what is happening. Not interesting.

For example:

  curl https://tuscolo2026h1.skylight.geomys.org/tile/names/000 | gunzip

EDIT: that's the flip side of supporting HTTPS that's not well-known among developers - by acquiring a legitimate certificate for your service to enable HTTPS, you also announce to the entire world, through a public log, that your service exists.

"I minted a new TLS cert and it seems that OpenAI is scraping CT logs for what I assume are things to scrape from, based on the near instant response from this:"

The reason presented by the blog post is "for what I assume are things to scrape from"

Putting aside the "assume" part (see below^1), is this also the reason that the other "systems" are "scraping" CT logs

After OpenAI "scrapes" then what does OpenAI do with the data (readers can guess)

But what about all the other "systems", i.e., parties that may use CT logs. If the logs are public then that's potentially a lot of different parties

Imagine in an age before the internet, telephone subscriber X sets up a new telephone line, the number is listed in a local telephone directory ("the phone book") and X immediately receives a phone call from telephone subscriber Z^2

X then writes an op-ed that suggests Z is using the phone book "for who to call"

This is only interesting if X explains why Z was calling or if the reader can guess why Z was calling

Anyone can use the phone book, anyone can use ICANN DNS, anyone can use CT logs, etc.

Why does someone use these public resources. Online commenter: "To look up names and numbers"

Correct. But that alone is not very interesting. Why are they looking up the names and numbers

1.

We can make assumptions about why someone is using a public resource, i.e., what they will use the data for. But that's all they are: assumptions

With the telephone, X could ask "Why are you calling?"

With the internet, that's not possible.^3 This leads to speculation and assumptions. Online commenters love to speculate, and often to make conclusions without evidence

No one knows _everything_ that OpenAI does with the data it collects except OpenAi employees. The public only knows about what OpenAi chooses to share

Similarly no one knows what OpenAI will do with the data in the future

One could speculate that it's naive to think that, in the longterm, data collected by "AI" companies will only be used for "AI"

2. The telephone service also had the notion of "unlisted numbers", but that's another tangent for discussion

3. Hence for example people who do port scans of the IPv4 address space will try to prevent the public from accessing them by restricting access to "researchers", etc. Getting access always involves contacting the people with the scans and explaining what the requester will do with the data. In other words, removing speculation

https://letsencrypt.org/2025/12/02/from-90-to-45#making-auto...

https://github.com/joohoi/acme-dns

Also, may I know which DNS provider you went with? The GitHub issues pages with some of the DNS provider plugins seems to suggest some are more frequently maintained, while some less so.

I'm also pretty content accepting the unpleasant parts of reality without spin or optimism. Sometimes the better choice is still crappy, after all ;) I think Oliver Burkeman makes a fun and thoughtful case in "The Antidote: Happiness for People Who Can't Stand Positive Thinking" https://www.goodreads.com/book/show/13721709-the-antidote

But what GP probably meant is that OAI definitely uses this log to get a list of new websites in order to scrap then later. This is a pretty standard way to use CT logs - you get a list of domains to scrap instead of relying solely on hyperlinks.