Pi-hole: A black hole for Internet advertisements
pi-hole.net
pi-hole.net
Super easy to install, full-featured, lots of lists to pick from, auto-updates lists, no need for an additional device, and you will benefit from router features produced by the openwrt community and maybe unavailable in your router proprietary firmware. Much recommended.
If that sounds attractive and it sounds like a good opportunity to change your crumbling unpatched router, the question "what's today's good cheap router running openwrt without trouble?" is frequently answered by https://www.reddit.com/r/openwrt/ :) .
I use Pi-hole now, and it works great. The one feature that I use quite a lot is the ability to disable it for a short period of time -- when I'm shopping for something, Google ads are sometimes actually useful!
I'd say I use that feature about once a month. That's the sum value of advertising for me.
I like to get my host list from https://github.com/StevenBlack/hosts
Did you check the 'Shopping' tab of Google search?
When you google search for something (like say a product or service that you want to purchase), you'll see a couple ads at the top of the results.
Pi-hole usually prevents most ads from even displaying, but those always show up. If the ad was useful and you want to click through, it will generally block the request. I think the click through sends you to doubleclick or something.
I've never disabled pi-hole in that situation, but there have been times when it was an annoyance because the ad was actually relevant.
But I presume Pi-hole has automatic updates to lists, data visualisations, better community support than the routers with open source firmwares which are often quite bug-ridden (not to belittle the effort though).
Those visualizations on the pi-hole look great indeed!
> "Pi-hole has automatic updates to lists, [...] better community support than the routers with open source firmwares which are often quite bug-ridden (not to belittle the effort though)."
OpenWRT/adblock auto-updates lists too, and I can't speak for DD-WRT or Tomato, but I've been pleasantly surprised by the support and quality I met using OpenWRT. About support, my few questions got answered quickly on their forum and r/openwrt. On the software quality side, apart from the UI being slow (which seems reasonable, it's running on a cheap router), sometimes complicated by lots of options (but at least they're available, and an effort is usually made to hide the exotic options under an "Advanced" tab), and blandly bootstrap-y, I don't remember hitting any bug.
Where as 100$ routers with factory firmware nowadays get security updates quite regularly.
[1] https://community.ubnt.com/t5/EdgeRouter/DNS-Adblocking-amp-...
If you aren't using it, you should!
Using Google DNS, self-hosted resolver, or your ISP's DNS: NSA, your ISP, everyone and every dog at the middle of your link to the Internet can track and see your requests.
Using CloudFlare's DNS w/ DNS-over-HTTPS: only NSA (via a NSL or subpoena), Cloudflare and CloudFlare's upstream can track and see your requests. And I guess 10%-20% of the domain names already use CloudFlare, so for some domain, it's end-to-end encrypted, nobody but NSA and CloudFlare can track you. Even better, Cloudflare is experimenting with peering to upstreams (e.g. Facebook) using private encrypted connections, so the point-to-point encryption ratio would be even higher in the future.
Therefore, using CloudFlare is a net positive.
But one also needs to consider its second-order effect: is giving CloudFlare more leverage over the Internet infrastructure in the long run an acceptable choice over unencrypted DNS? I guess everyone has a different opinion.
Follow up question, do you trust CloudFlare not to manipulate the results of DNS more or less than Google?
However be unable to determine which specific site you were accessing.
As opposed to Tor use, specifically?
a) Used where Tor is unacceptable, such as some university networks, and workplaces where using anonymization such as Tor/VPN is prohibited by policy.
b) When using Tor protecting yourself from the Tor endpoint collecting information / statistics on what you are visiting.
It seems to me like these DNS tricks are parlor tricks in a security sideshow. Any attacker that could see your packets can also see who you are connecting to. It's pretty rare that SNI does anything relevant to a real threat model.
I think a false sense of privacy is at least as dangerous as the alternative.
Yes they'd see that you're connecting to one of the largest reverse proxies in the world.
In any case, if you really want a full solution, build your own https://github.com/yegle/your-dns
Technically speaking the NSA wouldn’t be seeing your DNS requests, they would be seeing your ISP’s, for all its users anonymised.
If you use Cloudflare or Google DNS directly from home (or your own resolver), then yes, the NSA and anyone else can track your individual DNS requests directly.
In that regard using your own ISP’s DNS is clearly superior.
I judged the company in a negative light when their ceo or cfo wrote an open letter rationalizing their ban silencing some obnoxious website over political belief virtue signaling.
A company that crushes free speech cannot be trusted.
I don’t even remember what the obnoxious or offensive website was but I know that offensive speech is protected speech.
Autocratic technocracy centralized into a few digital monopolies wrap our wrists into digital slave chains labeled “free”.
Cloudflare is not the government. A business can choose not to service someone based on almost any criteria, that's not "crushing free speech". You can then choose not to patronize the business based on that policy. This is an important part of a free market.
Emphasis on 'claims', sadly we have no way of verifying that they don't actually keep logs.
https://community.cloudflare.com/t/1-1-1-1-does-not-resolve-...
In contrast, For my home network, it's just under 15% of queries that get blocked. I've got 3 Macs, a Windows 10 machine, an Apple TV (all connected 24/7) and a handful of iOS devices that hop on and off the network.
My blocklist contains ~114k domains I believe.
But as an adblocker - I feel like I'm missing something. It acts as a DNS server for your local network and blocks what's essentially a host file.
So how does it handle ads served through websockets?
How does it handle ads that come from the same domain as legitimate content (which is increasingly common)?
The complexity of rulesets by addons like ublock origin or PrivacyBadger seem to far surpass what PiHole is capable of.
I think PiHole has it's place on a network - obviously, but people have been promoting this thing like you can just get rid of your adblocker on your browser now.
People also downplay that this can be a pain in a home with a handful a streaming devices, each with a handful of apps. You end up whitelisting so much for those devices, you might as well whitelist the whole device just so the apps can work.
Your wife downloads a game on her phone, and you get that look like "ok, why isn't this working.. what did you do now?"
It just seems like a lot of effort for fairly imperfect results.
Sure installation is easy, but long term maintenance (the OS, the app, constantly whitelisting or troubleshooting when a new service or app breaks for someone in the house).
The pi-hole asks you to choose security over convenience, and you must accept that not all apps and services will work.
That's a personal choice you can make for your own setup in your own home.
I've never seen anyone say this _replaces_ your browser's ad blocker though.
This is an easy choice for myself.
it's not an easy choice for one person to make for a family of others.
I don't think it's a situation where you can ditch your ad blocker if you are dead set on never seeing an ad. It may be good enough for most people though. Personally, I still run ad blockers on my devices. Other people in the household do not.
I seldom have to whitelist anything. I may not have whitelisted anything at all. I have blacklisted a few extra domains - things like analytics requests for IoT devices. I don't recall a time that something didn't work and I had to fiddle with the pi-hole to fix it. It's been very low maintenance and very effective in my experience.
THISSSSSS. The only thing stopping me from using Pi-hole at home are my family members and the inevitable "this isn't working!?!?" rant and then I need to figure out how and what to whitelist. No thanks. I have ad blockers on the kids' PC and when something doesn't work, it's one click to temporarily turn it (browser extension) off.
If you have impatient shoppers in your household, the blocked affiliate(s) might be a benign issue at first, but when you miss out on a buying opportunity and then an algorithm prices it higher (while you are conducting a 'whitelisting' exercise) - things can escalate very quickly..
My rpi 3b (not 3b+) just couldn't handle it. It had 2 users. Our DNS resolution times increased by about 200ms. It was awful. I stripped it back out and haven't bothered trying to set it up again.
(Other details: the RPI was hardwired, wireless disabled, and it was a fresh raspbian install with zero customization outside of adding pihole.)
The router has a secondary dns server as well in case the rpi goes down (which has happened ~2 in the last year or so) or I need to fiddle with it.
It's good to know you guys haven't been having problems; I thought everyone was just fucking nuts or something, but no; local problem. Sigh.
I wanted to try it again and NOT do what I had done previously, but I think a conf file is still floating around because the second I install pfBlockerNG(maybe -dev too? I actually can't remember now), my entire network instantly goes down and won't come back until I remove the pkg again.
I don't know enough about BSD's package manager or where pfsense puts package conf files to try to track this down and stop clean it out. I'm sure I COULD figure it out, but I have other projects that are higher priority :)
Edit: I should also note when I was trying to figure this out I had a very angry spouse standing behind me burning holes into the back of my head because the network was down, so I didn't make a priority of really looking through logs and trying to properly diagnose things. I just wanted things to be up so that I wasn't slain.
https://news.ycombinator.com/item?id=20044430 [/tinfoilhat]
You set your DNS preference to point to the PI-hole and it should behave like any other DNS server. I guess it could attempt to resolve some spam domain like doubleclick.net and if it was incorrect it could complain...
There was a bit of controversy about FireFox doing exactly that. https://yro.slashdot.org/story/18/08/05/2353249/security-res...
Firefox is working on using DoH (opt-in at the beginning, but who knows) from "select" providers. Chrome has a similar switch, surely. Same with Android 9, opt-in DoH, but maybe it'll become opt-out or no-opt in the future.
In the name of privacy and security of course, but with the totally unintended side effect of users unable to dodge ads via DNS/hosts. Interesting, no?
It seems absurd that these companies are blurring the line between software and malware.
Software vendors are going to have to start paying for the bandwidth they use on networks against the will of the network operator.
Cloudflare for example would absolutely love to know what you're up to all day; and because they can now correlate data from their "omnipresent" WAF with the data from 1.1.1.1 they could get some interesting information... And believe you me, they're not sending it to /dev/null.
"1.1.1.1 does not mine any user data. Logs are kept for 24 hours for debugging purposes, then they are purged."
Are you claiming they are lying?
They could be forced to not talk about something via a gag order.
https://www.reddit.com/r/googlehome/comments/8917ci/google_h...
And so the arms race continues.
Paul Vixie got very upset when he discovered that his chromecast bypasses local DNS settings to go directly to Google: https://news.ycombinator.com/item?id=19170671
I wouldn't be surprised if soon Chrome defaults to DNS-over-HTTPs direct to base, except for the corporate intranet version. They just need to work out how to deal with wifi captive portals.
1) trusted endpoint / untrusted network (laptop in a coffee shop)
2) untrusted endpoint / trusted network (chromecast/alexa/other corporate zombie on your home network)
Which category a given scenario falls under depends on who you ask - to Google, Chromecast is in the first category. I don't know if it's possible to design a system that somehow always favors the rights of the individual.
This is why people keep objecting to technological solutions to social problems. Adblocking is a stopgap technological solution (although very effective at the moment); properly protecting the rights of the individual requires a social and legal process.
If the NSA has the capability to sniff vast amounts of network traffic, encrypting that traffic is a much stronger defense than telling the NSA they aren't allowed to deploy the capability for the time being.
If Chrome insists on using its own DNS or removing the adblocking API for add-ons, one can just use another browser like Firefox that has the desired technical capabilities. Managing DNS lookups and HTTP requests are not "stopgap" solutions, they are basic functionality that any one entity can't eradicate.
Alternatively, the traffic could be subject to heuristics to identify DoH connections.
(That would be a very annoying thing to do in a device or browser, but it's certainly possible)
I'm thinking about this, and feeling like the PiHole is a nice start, and I mean that sincerely, not sarcastically or dismissively, but what we need is a whole-house reverse firewall with that sort of capabilities, including everything the PiHole already does. If you did TLS interception, you could also pretty much implement uMatrix at the household level, for instance.
I should probably make an explicit point that I left implicit; I'm interested in anyone popping up and telling me "Hey, this thing exists already and it's http://...".
(I find myself wondering if I finally found my Rust project...)
The fact that this requires special hardware, bash commands, etc is severely limiting the audience. The more people blocking ads the quicker the internet changes.
Edit: thanks for the replies!
There was discussion a few days ago: https://news.ycombinator.com/item?id=20012687
There's a certain level of trust when I use 1.1.1.1 or 8.8.8.8. I'm unwilling to take the risk for this solution. I'm not sure what would help in the trust department to legitimize a solution like this.
Route hijacks can happen to anyone, even Cloudflare or Google. If anything they're more likely to be targetted than a smaller player like Nextdns.
I am not the relevant commenter, but what things would you expect to not change in a scenario where a majority of websites lost all ad revenue. (As admittedly unrealistic as it sounds the same was once true re moon landing and here we are, debating viability of not ruining our lives with advertising.)
Or could fund it with some targeted ads. Oh, wait.
I run my own knot-resolver server that forwards everything to 1.1.1.1 over TLS and I generate an .rpz that is basically the same filter list as pihole. Most DNS traffic ends up at Cloudflare, so you may as well go straight to the source.
https://gist.github.com/jzelinskie/3d2b11830224993fc8a7441b3...
include "/etc/bind/ad-blacklist";
/etc/cron.daily/update-ad-blacklist
or equivalent for unbound, maradns.... whatever.
I actually want to do the opposite: transition this to dedicated hardware (like a Pi, but worried about performance) that is a little less noisy. This is shockingly quiet for a 2U but I am a stickler for silence.
Some pix: https://imgur.com/a/0xwcfNN
I suppose for their whitelist/blacklist to work with regex matching the ads would have to be served from a similarly named domain. Like facebook.com vs ads.facebook.com, and you'd have to whitelist *facebook.com. And if they were getting ads externally you'd have to whitelist those ads for every site that you visit.
I regularly try browsing without adblocking on, and it's a constant nightmare. If sites held their advertising networks accountable to any reasonable set of standards, they wouldn't be in this situation.
The content I appreciate, I have found ways to support it.
If I go to a site and am bombarded with pop ups, auto playing video ads, etc., then yeah why wouldn't I block them? With the malware and tracking that is often injected into ads I have no problem using my adblocker at all times and disabling it for pages that ask politely.
I'm happy to click on ads on sites that I frequent and would like to support. I think there's absolutely a balance here, and for many years the advertising industry has abused their stay.
That's not my business. If they want to make money, they'll shift to other profit models that don't involve intrusive tracking and annoying advertisements.
The thing is, most articles are a distraction, a diversion. Something I do instead of the thing I should be doing and as such they are very very low value to me.
It became a problem when everyone and their sister started needing to know what kinds of kinks I'm into just to sell me dish detergent.
I've been predicting for a while now that sites would fall back to the old television show model "RockAuto presents MustangForums.com" or something to that effect.
Instead, we get a dancing Albert Einstein begging us to take IQ tests.
We post about office design and our ads are primarily for office furniture or other services related to the industry. We also self-host the ads which are non-animated jpgs and sell them without using any ad networks.
What you describe works well for us :)
I'm very interested in learning more...
People seem to have forgotten that users are most likely to be interested in what they are currently reading, because it is what their mind is currently focused on.
Because our content is so specific, ads which are relevant to the content end up targeting the user because you wouldn't be spending time on the site unless you care about the content.
It definitely takes time and work to develop and maintain relationships, but we also get to keep 100% of the revenue. In some ways we've just decided that ownership of the relationship and process is more important than being able to quickly slap some Adsense code up on the site.
That said, we also used Adsense early on, but have been doing this for ~5 years.
Minimal work involved. A bunch of emails back and forth, where we tell them our ad size if they provide an image. Or they provide a logo and we manually ad the text if they aren't technical enough to provide a custom image.
We also sell the space per month as opposed to based on impressions or clicks so it makes it a little more straightforward.
A couple things:
1. It facilitates a world in which only large content providers, who can afford to individually sell ads to advertisers, to exist.
There's a lot of overhead to ad sales and individual companies do not want to work with 1,000,000 providers, they want to work with 10-100.
2. It's substantially less efficient and only works for brand advertising or mass-market direct-response advertising.
One of the greatest things that Facebook and ad retargeting enabled was the rise of direct-response brands. Previously if you were selling a niche product - and most larger brands started out with a niche product - it was very difficult to reach an early audience who would be interested in purchasing your product. Facebook and Google flipped this on it's head, enabling millions of businesses to more efficiently reach customers. Facebook alone made the direct-to-consumer brand explosion we've seen over the last 10 years possible.
I am quite content to block these sorts of adverts and I'm not worried by the site's loss of revenue. I am not responsible for their slimy choice in business model.
Let's get back to context based adverts like DuckDuckGo use. There was no need for the internet to take this path - it only did so to rapidly monetise after the dot-com bust blew their VC funded rapid growth plans out of the water.
No need, aside from the trillions of dollars of economic value it created and hundreds of thousands of previously impossible businesses it created.
The flip-side of this is that I’ve noticed that YouTube shows me PSAs from my own municipal government (“there’s an election soon” ads, “we’re building a new piece of civil infrastructure” ads, etc.) I actually kind of like that; I don’t have cable, so it’s not like I would see them anywhere else.
The entirely-static ads model does work when the consumption of the media is entwined with the consumption of the advertised brands, though. For example, a podcast can certainly advertise its own tour, since—given that you’re listening to the podcast—you likely want to see the podcaster speak in person, even if you can’t make it there.
Or, of course, if a (global) website is just advertising another (global) website. The NYT can advertise Amazon just fine.
We fully control and host our static ads and try to keep them high quality, so I've decided that minimally using IP to loosely serve a more relevant ad is okay. MaxMind offers a downloadable Geo IP database that we use to do this and are not needing a 3rd party service for this.
The upside for the user is that location and whatever the one site is able to determine about the user is all that can be shared. If the user hasn't logged in with their real name - that probably isn't much.
So instead of one ad being enough to pay for your content, you have to fill your website with banner ads, embedded ads, scroll over ads, animated ads, etc etc.
It's a slippery slope, more people use adblockers causing content creators to add more advertisements to generate the same amount of income. More people are bothered by the increase in ads, and download adblockers themselves. Rinse and repeat until ad supported content is unrealistic for all but the biggest of websites.
And I'm pretty sure even checking location is controversial. I've at least seen it included as part of tracking in the past.
That's still possible with static ads. The server can simply lookup your country from your IP address and serve the relevant ad, without tracking you at all.
I don't have cable, either, but I do have a pair of rabbit ears to keep up with local news via OTA broadcasts.
[EDIT] to expand, I think the piles of spyvertising money funding sites & services is a big part of the decline of truly free sites and open protocols, and make running a paid site (or app, or whatever) harder since you're competing with "free" (but spying on you). Less incentive to use them, less incentive to contribute to them. The whole system's perverse and harmful and it would 100% not be the end of the world, or the end of nice things for free/cheap, if it just disappeared tomorrow.
It is not our responsibility to prop up their poor model. If these sites want to make their case that they won't survive without our eyes on their ads, then they can open their books for us to look at the costs and revenues and decide for ourselves whether or not we should help them. But at the end of the day, it is their problem, not ours.
In the world we live in something like a pihole isn't an ideological position, it's a necessity to not have everything we do end up rolling into someone else's ad profile on my household.
giantbomb.com sells premium subscriptions and merch and does okay
I choose who deserves my attention carefully. Internet ads have not earned it.
I do that for the sites that have the banner that hides if you scroll down, but pops out the moment you scroll one pixel back, or sites that put up "please don't leave me" modals the moment your cursor strays out of the window.
The answer being that content providers can't be trusted to self report metrics that determine how much advertises pay. At least not for pay per view/client/etc models.
The people that self select themselves from viewing advertisements might be doing advertisers a favor. They're perhaps less likely to make purchases based on impressions//click ads on purpose; per dollar, ad campaigns might be more effective without said people.
There isn't a chrome extension or anything to white list a site quickly. You have to go back into the interface, login, and whitelist, go back and load the page then you'll find that you needed to whitelist a few subdomains/cdns as well. This is really fun when you've got all your devices using the Pihole for DNS and you can't load something on your phone/TV and need to run to your laptop to deal with it.
If you just got your pihole you probably threw in a bunch of community generated lists and you'll find a good amount of stuff you do visit gets blocked. You can get to Google but not Google drive, so you whitelist it. And you do this over and over again until you finally get annoyed because you just want to make a car payment so you permanently disable it for 5 minutes, or 60 minutes if you've gotten annoyed enough.
Sometimes weeks will go by and you'll forget you even had it disabled at all.
FWIW, I also don't use NoScript because I find it incredibly annoying. This is one step further from the NoScript annoyance because you have to go into the webUI and make your changes.
If you don't mind NoScript you'll probably be fine with Pihole. Or if you have the time to curate and pick lists that fit exactly within your browsing habits.
I know it doesn’t sound very sentimental, but the first time I showed my relatives what the Internet looks like without ads, I think those were the strongest hugs I ever got from family members.
I commented on a different post last night, that I was a bit shocked and saddened to see their Patreon is only pulling in $1,700/mo.
Do they have another significant revenue stream? Is it just too much hassle to bother signing up to Patreon to commit to even $1/mo? Do they have something on the Admin panel where users can click to pay directly?
I’m not judging, I don’t even have a Patreon account. I’m curious how such an apparently crucial and useful piece of software — one that no doubt is responsible for providing millions of dollars of value to its users, and perhaps blocking tens of millions of dollars in ads — how can the project be sustainable after 53 releases and 2,700 issues on Github while pulling in less than $24k/yr?
This is an astonishingly huge amount of money for an open source project to raise directly from its users. Most open source projects get basically nothing.
First, certain streaming websites would fail and it was too much trouble to try to find the URL to whitelist.
Then after I had disabled it from the Pi-hole interface everything was fine but it wasn't actually active. No problem...until I forgot my router was using it as a DNS server and I moved and didn't set my Pi up yet. Then it took me a couple weeks going back and forth with Comcast to find out that my router was still pointing to a DNS server that wasn't running.
Somehow my FireTV bypassed the bad DNS server at one point (still no idea how this happened cause my router was routing all traffic through the IP for pi-hole) and that made me realize that I can get data from Comcast somehow so maybe it really was my router.
Also beware as most ads in your phone apps come from ad intermediaries that are either dynamic or constantly change.
Pi-Hole is a cool project but please take in consideration those two when using it. We are far from the 90's in ad-tech.
> most ads in your phone apps come from ad intermediaries
I don't know about the intermediaries you are talking about, but all the ad-ridden proprietary mobile apps that I use (the ones that don't self host ads) are blocked by DNS based ad blockers.
The one thing that these DNS based ad blockers can't do however, is block in page annoyances which is why using an extension like uBlock Origin is still necessary.
Anyway good luck with that if the app is using a mediator from a big known name as it will likely block all of their services as well.
it worked pretty well
[] Root user check
\e[1;32m.;;,.
.ccccc:,.
:cccclll:. ..,,
:ccccclll. ;ooodc
'ccll:;ll .oooodc
.;cll.;;looo:.
\e[1;31m.. ','.
.',,,,,,'.
.',,,,,,,,,,.
.',,,,,,,,,,,,....
....''',,,,,,,'.......
......... .... .........
.......... ..........
.......... ..........
......... .... .........
........,,,,,,,'......
....',,,,,,,,,,,,.
.',,,,,,,,,'.
.',,,,,,'.
..'''.\e[0m
[] OS distribution not supportedI'm using this[1] but I'm surprised there isn't something more official/baked.
Anybody familiar with this code able to point out where it does the "interesting" work?
>root@pihole:~# uptime > 17:02:51 up 587 days, 22:34, 1 user, load average: 0.03, 0.03, 0.05
Explain your setup some more and I can add more details.
https://discourse.pi-hole.net/t/change-upstream-dns-server-i...
If you're intending to use OpenVPN, you could easily justify a basic x86 pfSense or linux router: https://arstechnica.com/gadgets/2016/04/the-ars-guide-to-bui...
I run both pi-hole and my own DNS server inside my network as containers on the NAS. I then have my router configured to default to the pi-hole and then the DNS server.
Advantage of my own DNS server is it exclusively resolves using DNS-over-TLS so my queries are private.
Final fallback for resolution is 1.1.1.1 but based on logs my setup hasn’t hit the fallback.
I imagine you could also use a container to host VPN.
tmpfs /var/log tmpfs nosuid,nodev 0 0
anyone have this type of setup?
Previous discussions =>