Hacker News Reader

Comment by warent | Hacker News Reader

I run a network ad-block dns (pihole) and consistently 25-33% of all my network traffic is blocked as ads. It's much more than I ever imagined. Now I'm used to a different internet, when I'm using internet off the network it's like WTF is this?

View on HN

25-33% of requests? Or is this a percentage of bytes?

Because I wonder what percentage of bandwidth (in terms of bytes) trackers/banners/ads account for.

Need to set up a pi-hole ... just too many other projects....

Pi-Hole is a DNS solution so it's just blocking DNS lookups. Mine is currently blocking 43.9% of all DNS requests.

It's a percent of DNS requests. It might be quite difficult to see what percentage of bytes it translate to, since HTTP requests aren't actually sent.

My pihole is showing 18.7%-23% of requests blocked :)

I set it up recently. It's about as much effort as buying and setting up a new laptop with ubuntu if you get a kit. I'd imagined it as a project beforehand, but in reality it's super easy and trivial (assuming you're comfortable using linux and ssh at a noob level).

Yep same. I'm a mere web developer that mostly works on Mac. Getting Pihole setup only took me like an afternoon after having an RPi sitting around doing nothing for months. They make it really easy, just follow the instructions. Also I'm lucky enough that my router has a friendly interface where it's easy to set the router DNS to pihole.

How would you measure bytes if the requests are blocked?

Some people are good at thinking a process through to the end. Others are not and ask questions at the first unknown. It's a large part of why I'm not a teacher.

Run the same requests throught different end points. Each through pihole & unfiltered, while monitoring the traffic on both.

Doesn’t that defeat the point of pihole? Though I suppose if what you want to do is measure things it makes sense.

Yeah, that's what I was thinking. Just to get a sense of how much bandwidth I am saving by in fact blocking the request.

It would be nice to know how much we're wasting.

I just have a decent set of ad blockers and the experience is similar. Unfortunately, it often results in weird experiences or I get sites which don't work at all if you have ad blocking.

Well that's interesting. For me 99.5% of websites work perfectly using ublock origin. The only .5% remaining are websites that actively refuse to serve any kind of adblock users, not because it breaks functionality on their site. I don't think I can recall having visited a single website that would have features break unintentionally because of ublock in the past few years.

Can't agree with you, Dynamics 365 is one of them (it's shit but I've implemented it at work). EDF (French main electricity provider) also breaks for me. That's one example from a big company, and one example with a big user base.

It depends on how many privacy lists you have added, probably.

Normal display ads all being blocked is generally fine 99% of the time, but if you care about not being permanently tracked across the internet then there are a couple more domains you have to add - except some sites make it mandatory that those invasive fingerprinting scripts and port scanners run and report back a session, otherwise you're refused login or banned.

I don't think my experience is vastly different from yours. I do get some sites where pop-overs or cookie notifications are blocked but it's not clear and you just can't scroll. I could turn off those blocker settings, but the notifications are annoying enough it's worth it.

The Denver Post just lost my business over this. They have one of those things that scrambles all the words for any user with the audacity to not want to see video+audio ads while reading their newspaper.

Is it their content to do what they want with? Sure.

Does the same logic apply to the $9 I used to give them each month? You're damn right.

Hard agree. If I'm paying for content, I'd accept a small number of discrete advertising. Video advertising on a text/ photo site pisses me off in general and if I was paying for it? No chance I'd let that fly.

Pi-Hole/NextDNS also blocks adds in most apps. I used NextDNS (which has a limit on the free tier), and recently switched to pihole running on my home server. I also use ZeroTier to connect to my server directly even when I am not on my local network to still use it as the DNS server. Works great.

I paid them, $20/yr is quite good and I can add my parents' house, my in-laws' house, etc on the same plan and manage them all centrally.

You can get even better coverage with the NoTracking lists (dnsmasq/unbound or dnscrypt-proxy) https://github.com/notracking/hosts-blocklists

They focus not only on tracking but also malware prevention, where possible via dns filtering.

Pi-Hole still does not properly support wildcard filtering, only via regex but that is not really efficient (requires tons of resources).

Question about pihole: is it possible to turn off blocking for a website? Do you have to log into the pihole web interface to do that? I often go to websites where some crucial functionality is blocked by my adblocker (ublock origin), where I have to turn it off for that site.

Yes, you can do that via whitelist/blacklist: https://docs.pi-hole.net/guides/misc/whitelist-blacklist/

You can whitelist yes, or there's an option to disable the entire thing temporarily for x minutes.

Yes you have to log in to the interface unless you engineer a way around it

On iOS there is an app called piHoleRemote that had a nice widget that allows you to disable pihole for x minutes.

Can be nice to use to quickly disable pihole to get through to a particular website.

Yes, you have to login to disable but you could easily use the API. For instance, pihole.disable(60) with https://pypi.org/project/PiHole-api/

I just stop using sites that gimp themselves when I use an adblocker. There's tons of alternatives for most things.

How do you use the web when you can't click on links?

I can't effectively keep a mental black list of all the sites which I don't want to click on.

Very few sites are broken with ad blocking. If you click on one, you just press the back button. No need for a mental blacklist.

I don't. I mean, if it's a news site just search for the title in a search engine and you'll find other articles. If it's a web application I search for an alternative and bookmark that. If you really want to avoid even loading it, you can just block the whole site with your adblocker but I don't go that far.

That is what I currently do. It turns casual browsing into a frustrating scavenger hunt. The whole point of the web was to make links effortless so you could browse sites. This breaks that whole model.

There aren't always alternatives - think shopping for certain items, government forms.

Why would a site that hopes you'll send them money in exchange for product, refuse your traffic if you have an ad blocker enabled? That just costs them money. Same for government forms, why would they refuse your traffic if you're blocking ads?

I wonder, too. Yet I still see these issues.

It might not be intentional to break the site experience for adblock users - but there is a number of sites that has implemented link tracking in a way that overrides the normal click (though sometimes not keypress) events, to let the tracking code do its thing. If the tracking code is blocked or fails to load, that means a lot of actions break.

Best part? Trying to convince the operators of such sites that users they cannot see in their "analytics solution" are worth fixing their site for is not exactly a straightfoward job - from their narrow view, these users simply do not exist, because the tracking does not show them!

You have adds on your government forms?

Ad blockers have false positives.

It is not what GP asked.

An no, there isn't "tons of alternatives". In theory there is. But in practice, they can really make your life harder. Some may say that Signal is an alternative for WhatsApp, but if people you communicate with don't want to use anything but WhatsApp, then Signal is useless. I hate Facebook but when I want to plan an event, I found nothing better, simply because that's the platform that reaches the most people. Network effects... But also, your favorite show may not be on "alternative" streaming platforms, sometimes your job, or worse, the government may require a specific website.

There are extremists who are ready to find alternative friends, shows or jobs just to avoid using some website. It is a good thing these people exist, that's how progress is made. But for most people you have to make compromises.

> I hate Facebook but when I want to plan an event…

Ah! That's why I haven't missed Facebook. I am old enough that I don't plan events any longer.

(Or maybe I have no social life. Actually, that's right, I don't. ;-))

My solution to this is using Cloudflare Warp (Cloudflare's consumer-facing VPN).

When I need to access ads.google.com or analytics.google.com for my company, I turn on Cloudflare, and pihole is bypassed.

I map the command below to a keyboard shortcut to disable all pihole blocking for 60 seconds via the pihole disable API call.

wget --quiet "http://PIHOLE_IP/admin/api.php?disable=60&auth=YOUR_API_TOKE..."

You can find the token in the pihole Web GUI at, Settings > API/Web Interface > Show API token

I run Pi-Hole on my network as well; it's wonderful. I'm terrified that it will stop working soon though, as companies start to use their own DNS servers, which I've heard is happening.

Interesting. I would think though that the move to their own DNS servers could extend to their own ads as well — that is, cutting out the middle man that is Google/etc.

I'm all for news sites, for example, hoisting ads if I knew they were getting the money from those ads, knew the ads were actually coming from their site.

Sorry, I didn't mean their _own_ servers, I just meant hard-coding 8.8.8.8 into the DNS settings, for example.

I wonder if you could hijack those requests at your router and send them back to your Pi-Hole? But then they just switch to DNS over TLS...

I just have my network block outgoing DNS queries that aren’t from the gateway. But you’re so right that DoH is going to throw a wrench in this.

If an ad can use DoH to sidestep a firewall, so can an employee. If Google and Facebook were cunning (and nefarious, but that much is presumed), they would be aggressively developing a product that solves this problem for corporate networks, but at an enormous cost. Otherwise, when corporate networks solve this (and they will), home users who hate ads will just follow whatever pattern they settle on.

Encrypted DNS is going to upset a lot of corporate network management/monitoring... But the switch to HTTPS for everything caused similar frustration for IT admins awhile ago.

In the corporate world, I think the future is managing your network by managing every single device on your network. Only let authorized/corp devices in and all those devices must be enrolled in an MDM solution that enforces all sorts of policy and includes monitoring traffic/DNS queries. Of course that's a lot more work than just monitoring things at the network level.

This is pretty much why Google are a huge proponent of DoH.

I'm not sure I understand how this is related to pi-hole.

Your computer sends the raw domain name to pi-hole (e.g. ads.google.com), and pi-hole returns 0.0.0.0 if it's on the block list.

There's nothing Google or anyone can do to make pi-hole stop working.

IoT devices are starting to use hardcoded DNS servers instead of using the one provided by DHCP, which negates the benefits of Pi-Hole on those devices.

For now, I've configured my router to force all UDP port 53 traffic to my Pi-Hole which overrides what I mentioned above.

But, in the future we may start to see IoT Devices hard-code DoH servers which will be harder to force over to the Pi-Hole.

> I've configured my router to force all UDP port 53 traffic to my Pi-Hole

Oh wow, how do you do that?

Would you mind sharing the blocklists you use? I have gotten to a ratio like that, but I have noticed that it was causing more issues with regular websites for my guests, so I removed many of the custom ones. I'd like to try some others if you have suggestions.

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [] Status: Retrieval successful
  [i] Received 59896 domains

  [i] Target: https://mirror1.malwaredomains.com/files/justdomains
  [] Status: Not found
  [] List download failed: using previously cached list
  [i] Received 26854 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
  [] Status: No changes detected
  [i] Received 34 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  [] Status: No changes detected
  [i] Received 2701 domains

  [i] Target: https://dbl.oisd.nl/
  [] Status: Retrieval successful
  [i] Received 1167690 domains

  [i] Target: https://phishing.army/download/phishing_army_blocklist_extended.txt
  [] Status: Retrieval successful
  [i] Received 21379 domains

  [i] Target: https://raw.githubusercontent.com/deathbybandaid/piholeparser/master/Subscribable-Lists/ParsedBlacklists/AakList.txt
  [] Status: Retrieval successful
  [i] Received 5 domains

  [i] Target: https://raw.githubusercontent.com/deathbybandaid/piholeparser/master/Subscribable-Lists/ParsedBlacklists/Prebake-Obtrusive.txt
  [] Status: Retrieval successful
  [i] Received 3 domains

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
  [] Status: Retrieval successful
  [i] Received 14724 domains

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
  [] Status: Retrieval successful
  [i] Received 412 domains

  [i] Target: https://raw.githubusercontent.com/hectorm/hmirror/master/data/adaway.org/list.txt
  [] Status: Retrieval successful
  [i] Received 9182 domains

  [i] Target: https://raw.githubusercontent.com/hectorm/hmirror/master/data/disconnect.me-ad/list.txt
  [] Status: Retrieval successful
  [i] Received 2701 domains

  [i] Target: https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txt
  [] Status: Retrieval successful
  [i] Received 209608 domains

Noticed that some apps on iOS spammed the DNS server if they couldn't connect to their ad networks which should affect battery negatively.

I added this to my /etc/hosts

https://github.com/StevenBlack/hosts

What is the advantage of having DNS on a separate device other than that it provides ad blocking for multiple devices?

That’s the main benefit.

But also you can have more flexible block patterns. I run DNSCrypt-Proxy and my block lists can have wildcards. With /etc/hosts you have to enumerate each origin. It can also do things like IP blocking where if any domain resolves to a known ad network IP, then that request is blocked.

But mainly, DNSCrypt-proxy encrypts all my outgoing queries and round robins them across resolvers. (Also hi dheera!)

I use the noscript extension that uses a scripting whitelist. Bit of a pain at first, but pretty soon your browser will be flying. No extra hardware needed.

Kid's computer has dnsmasq as a similar solution.

My issue with pihole or any other DNS adblocker is I can't whitelist some website that I love. As evil as ada network, but I still want my favorite site to get some revenue.

Maybe your favourite site would welcome a direct contribution, instead of an ad click

Mine was that high, until I ditched all the family Android devices. It's now around 2-3%. It's quite an extraordinary difference.

I use NextDNS for this, it's brilliant. (I'm not affiliated with them in any way, just a happy customer.)

Same. I got tired of Pihole breaking for one reason or another (although I certainly adore the project). NextDNS works extremely well, provides a native app for every device, runs on my router, and is dead simple to maintain.

Hear hear. The only problem I’ve had was when I set it up on my router and my IKEA smart lights stopped responding after some 30 min or so. Turns out the gateway phones home and those calls were blocked, so for some reason or another the gateway just stopped responding to commands. Restarting it or resetting the network made it fly again, but only for the set time before it phoned home again. I was very disappointed by that, after having read some article here on HN arguing that IKEA had actually done IoT sort of right. Oh well.

Obviously not a NextDNS specific issue, it’d happen with anything that blocks the call, but just putting it out there for the next sucker that tries to google why their IKEA gateway suddenly stops responding.

Would Pihole affect latency in online games?

Reply on news.ycombinator.com