TuM'Fatig – Ads blocking with OpenBSD unbound(8) | Hacker News Reader

TuM'Fatig – Ads blocking with OpenBSD unbound(8)

tumfatig.net

59 pointsprotomyth29 comments

> Grab and parse the Top 10 milion domains (based on Open PageRank data) so that they can be used by dnsperf(1).

Be vary of using such sources.

Go to the source [0], make "Display: 100" and scroll to "1,351 to 1,450 of 10,000,000 websites".

You will find a bunch of bullshit domains clearly used for something nefarious (and some no longer exists), but to believe what those sites are as popular as spreadsheets.google.com? NO WAI

Tiny sample for the lazy:

    1,361    ecoquipo.ga              5.92/10    ga
    1,362    h4uvpev4tx.ga            5.92/10    ga
    1,363    lfsni36qvn.ga            5.92/10    ga
    1,364    dashseo.ga               5.92/10    ga
    1,365    velocityconf.com         5.92/10    com
    1,366    filezilla-project.org    5.92/10    org
    1,367    bx2vnequhc.ga            5.92/10    ga
    1,368    incblizzard.ga           5.92/10    ga
    1,369    anisima.ru               5.92/10    ru
    1,370    ou14ib22tf.ga            5.92/10    ga
    1,371    i12tfukfft.ga            5.92/10    ga
    1,372    blankmedia.ga            5.92/10    ga
    1,373    nicelia.ga               5.92/10    ga
    1,374    finlandtoday.fi          5.92/10    fi
    1,375    spreadsheets.google.com  5.92/10    com

[0] https://www.domcop.com/top-10-million-domains

So I've always been curious, does anybody have a writeup of how DNS servers like unbound etc are able to handle evaluating these massive blocklists? Like the algorithms and data structures they use to evaluate queries against these lists efficiently. My pfsense+pfblocker unbound instance has something like 400K hosts blocked and I've always been amazed that it doesn't even break a sweat scanning that many hosts with every query.

Here's Tony Finch's discussion of the data structures he's working on for ISC's BIND:

https://dotat.at/prog/qp/README.html

> My pfsense+pfblocker unbound instance has something like 400K hosts blocked

But it is not 400k hosts to search:

    com
        facebook
                 www
                 ad
        google
               ad
               ads
               notadswepromise
        googletagmanager
                         tags
                         ads

to block ad.facebook.com you only need to scan 10-100k of .com domains in your block list and only a couple of records in facebook.com. Don't forget, DNS is a hierarchical system. And we, humans, actually record, use and remember all the DNS records in the wrong, reverse direction ;-)

> 400K hosts

I bet 30% of those hosts are don't resolve anymore and another 30% resolve to domain parking/404 hosts.

Eg: see my comment about "TOP 10M SITES"

https://news.ycombinator.com/item?id=33150554

I bet 30% of those hosts are don't resolve anymore

I've seen some evidence of this. I periodically dig all the blocked domains and noticed that some of them come and go so I assume people cycle them in and out to evade blocking. A handful of the domains just change the A record so I just block the apex domain which makes it a little harder to keep the block lists optimized but does reduce the size a little.

I should add that blocking the apex of some domains includes risk of dropping legit traffic but if I see more than {n} badware domains on the apex, I write it off as compromised or at very least poorly implemented and poorly secured.

I suspect parent meant the way the list is processed and how the structure resides in memory.

We'd optimally want, e.g., aaa.ai to resolve as fast as zzz.zx. If we just loaded the file for every query and ran down the list linearly we'd probably see a large disparity with the former being much quicker than the latter.

So, the question would be: how is it done in an efficient way? Is it loaded into a btree, is each domain broken into various parts, are certain domains prioritized, threading, compression?

Plus, there's the queries for dns updates, and how the results get integrated.

Some combination of tries, B-trees and Bloom filters would get you most of the way there.

https://news.ycombinator.com/item?id=20148457

Not sure how out of date that is

This is really cool, thanks for the write up. I did the same thing in my homelab but much simpler, my shell script only fetched from a few sources and converted it into unbound format so it was much shorter. And I didn't care about statistics, but the idea was the same.

Also, why use ftp instead of curl?[1]

1. https://curl.se/docs/manual.html#time-conditions

My guess is that curl is a port/package while ftp comes with the base system.

True but I also run OpenBSD at home and curl is part of my common packages to install list in ansible. How I bootstrap all my nodes.

As others have mentioned, ftp that ships with OpenBSD is a bit more robust then its name suggests. https://man.openbsd.org/ftp

The ftp client is, by default, installed in the OpenBSD base system. Even though the name is ftp, it works with HTTP, HTTPS, FTP/FTPS and so on.

is it like poor man's curl?

This is what I do. I was trying to figure out what pi-hole did one easy thing was it black-holed domains, so I got the same list put it on my dns relay box and there you go same thing on openbsd.

https://pgl.yoyo.org/adservers/

basicly all it is is

     include: /var/unbound/etc/yoyo.conf

where yoyo.conf is a dns blacklist

Glad you're finding the list useful!

Just for the record, but my list is available natively in a format for use with Unbound:

- https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound

Remove the header with &showintro=0 and for plain text, use &mimetype=plaintext:

- https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound&mi...

As a fun experiment one could set this up on a handful of VM's and leave the Unbound TLS port open or DoH if compiled with libnghttp2 to the world in addition to the graphs. I think it would be interesting to see how many clients each VM could handle. Maybe enable the rate limits in Unbound first. I'm half tempted to do this with a few of mine but I do not have the fancy graphs, just shell scripts to parse the logs which I guess could be written out as text files for nginx to serve.

Unfortunately for me the Unbound package on Alpine is not compiled for DoH, only DoT and I am too lazy to recompile things any more.

What is TuM'Fatig - klingon or something?

It's french slang, = Tu me fatigues = "You make me tired" (tired of ads I assume)

A more correct translation would be "you wear me out".

This is but one post on his blog, though.

means, you ~ irritate me

In spite of my entire satisfaction with it, I had to ditch my OPNsense router... Its unbound ad filtering DNS is among the features I miss most from it.

Why did you have to ditch?

Bus factor. I had a very nice home setup (three 24 ports switches and a 6-port router) with multiple VLAN, GPON terminated on my own device (emulating the ISP's hardware), bonded Ethernet links, fully routed IPv6, sane firewalling, ad-filtered DNS, reverse proxy etc. - but I was the only person in my home (six kids + adults) to know how to maintain it. Anything happens while I'm away and Internet access stops. I borked an update and was offline for a couple of days - that was once too many for the users.

So we are back to the ISP's router, no VLAN and soon some Wireguard overlay for my hosting.

I'm sad about this: replacing infrastructure I control with the ISP invading my home, replacing tailor-made with generic consumer products, replacing native Internet with yet another overlay network... But MTTR rules my world.

I just realized that OpenBSD unbound was likely named as a counter to ISC BIND. Similar to Sun Java and the Eclipse Foundation.

It is not really openbsd's unbound, it is nlnet labs unbound. openbsd just ships it with their operating system.

https://nlnetlabs.nl/projects/unbound/about/

but yes it was made to avoid a bind monoculture. and comes in two parts nsd is the authoritive name server. and unbound is a recursive name server.

As somebody who spends some time on this, identifying common infrastructure and targeting that is the best bang for the buck: all kinds of stuff lives in the sewer that you never thought of. OTOH it does require some whitelisting, because all kinds of stuff lives in the sewer.

Reply on news.ycombinator.com