Hacker News Reader

Comment by Eikon | Hacker News Reader

Shameless plug: Check out my Certificate Transparency monitor at https://www.merklemap.com

The scale is massive, I just crossed 100B rows in the main database! :)

I'm clearly not the target audience for this, so excuse me if this is a dumb question: what is this tool used for? Who would usually use it and for what purpose?

Anyone setting up infrastructure, security researchers, security teams and IT teams.

It’s also actually very useful too in the brand management field, especially to detect phishing websites.

Are you continuously monitoring consistency proofs? Or in other words, would someone (you or someone else) actually notice if a log changed its contents retroactively?

Not yet, but that’s definitely the short term plan!

Why do it only show a few subdomains for .statuspage.io? I would have expected at least 10K or so. https://www.merklemap.com/search?query=*.statuspage.io&page=...

Is my query wrong or are you just showing less results intentionally if you’re not paying?

> Why do it only show a few subdomains for .statuspage.io? I would have expected at least 10K or so. https://www.merklemap.com/search?query=*.statuspage.io&page=...

Because they have a wildcard for *.statuspage.io, which they are probably hosting their pages on.

> Is my query wrong or are you just showing less results intentionally if you’re not paying?

No, results are the same but not sorted.

I tried to do something like this one time and had a problem just finding the logs. All information on the internet points to the fact that certain logs exist, but not how to access them. Are they not public access? Do you have a B2B relationship with the companies like Cloudflare that run logs?

They're required to be public services. https://crt.sh/monitored-logs is the list of logs monitored by crt.sh (a public log monitor operated by Sectigo, a commercial CA) if that helps. Each of the major browsers also publishes which logs they trust and provides information about e.g. distrust of logs. Is the problem that you couldn't figure out how to use a log? It doesn't just have a web site where you can type in searches you need to be able to use their web API as defined in the protocol documentation.

This is really cool! It discovered even subdomains that lived for a few days on my site. If it’s not a secret, how do you discover those? Is it by listening to DNS record changes?

By using... certificate transparency logs.

https://www.merklemap.com/documentation/how-it-works

That's interesting, can you share more information about your tech stack?

Merklemap is running PostgreSQL as the primary database, currently scaling at ~18TB on NVMe storage, and around 30TB of actual certificates that are stored on s3.

The backend is implemented in Rust (handling web services, search functionality, and data ingestion pipelines).

The frontend is built with Next.js.

Why not use something like crt.sh?

Why not use something like Merklemap?

Reply on news.ycombinator.com