Secure and Ad-Free Internet Anywhere with Streisand and Pi Hole | Hacker News Reader

Secure and Ad-Free Internet Anywhere with Streisand and Pi Hole

ifelse.io

306 pointsmarkthethomas108 comments

>I recently took some time off of work to spend time with family. During that time I worked on a few side projects like getting our backups in order and getting a Raspberry Pi 3 running Pi Hole.

I know what the original intent is, but still- I found this quite humorous.

Still, this seems like a very nice noob-friendly guide, and that's a very good thing when VPNs are concerned.

hah! I meant "in my free-time" lol but glad you enjoyed!

Thanks, nice post. Also enjoyed your write up about backups: https://ifelse.io/2018/12/05/better-faster-more-secure-backu...

The funny thins is that I’ve been thinking something similar during Christmas. I now know that I’m not the only one thinking about taking days off work to sort out my backups and network :-)

awesome! glad you enjoyed it :) trying to write up the things I find super helpful (code-related or not) and share them. especially when I can't find any good docs on how to get something working lol

I run a pi-hole equivalent on a Raspberry Pi at home (dnsmasq with pi-hole's block lists). One word of warning, its list tends toward over-zealous, blocking sites / links you might want (e.g. Google Shopping links). You can add manual domain exceptions but it's tedious and takes a long time to restart dnsmasq due to the enormous list size and the RPI's relatively slow performance.

Blocker extensions are nice becuase you can selectively disable them or open an incognito tab in cases when you need to bypass the list. Easier than temporarily changing your DNS server to 1.1.1.1, etc.

Still, nice to have automatic adblocking for all devices in the house.

Ideally, one would hope a Pi-Hole would just block Google altogether. ;) Iunno why you'd want Google Shopping links to work.

Jokes aside, it'd be nice if there were easier ways to mark certain groups of ads as okay. For instance, if I'm paying for a streaming service at the "comes with ads" tier, I'd actually prefer to allow those ads, without having to figure out which ad domains Hulu is using.

The biggest reason to go for the network level block though is convenience across many devices, as you said, as well as blocking tracking and ad domains that aren't running in your web browser: Apps themselves now come with ads that tend to be a lot harder to block on the device.

> Iunno why you'd want Google Shopping links to work.

I don't know why you'd want links that you intentionally click to be broken. ;)

The problem is not the links you click, it's the links that load nasty stuff during page load.

This is why a rewrite proxy is better than a simple DNS blackholer. Unfortunately a side effect of https ubiquity is that rewrite proxies are much more complex to deploy, even on your own local machines.

I do. I use a hosts blocklist that blocks some inline trackers, like affiliate redirects you find on SlickDeals. I accidentally click those frequently so I want them blocked.

I may be silly but I want the affiliate links I click on Slickdeals to work.

I use pfBlockerNG (which is basically just a pi-hole equivalent that integrates with pfSense) and I only use the Steven Black list:

https://github.com/StevenBlack/hosts

It seems to be well curated. Breakages are rare, and if there is one, you can file a bug.

Works well.

This is exactly the information I came in here looking for. Thank you.

I had the same problem with overzealous blocking. I had to disable temporarily about once per day, and my wife found it bothersome also (especially because she didn't know how to manually disable it). We turned it off last month and I've just been using Brave instead. Works great on my MBP (fan doesn't run nearly as much as with FF & Chrome), though of course our mobile devices aren't protected anymore.

If anyone has a good solution for the overprotectiveness of the Pi-Hole, I'd love to hear it!

Isn't the overprotectiveness of Pi-Hole in relation to the blocklists not Pi-Hole itself. I'd suggest findings different blocklists.

There is one particular website that is extremely popular, I can't remember what it is now but it has blocklists showing which are likely to cause false positives.

https://firebog.net/

Only add the ones with a checkmark.

I have no idea about how Pi Hole internals work being myself happy with Ublock Origin and other browsers extensions, but anyway a database storing the danger level based on IP nature (plain malicious, NSFW, advertising, etc.) and associated risks into multiple fields could make things easier for the user who for example could want to surf gambling or porn sites while keeping the guard high against phishing, advertising etc. A series of switch would be more immediate to use than editing a long list.

You should use the whitelist options.

You could just disable the pi-hole dns server in your routers DHCP server (so it will not be used by all network devices by default.)

Then, only on your own phone/laptop/etc, manually configure your pi-hole as the DNS server for the corresponding home Wifi network. So only the devices you manually configure will use the pi-hole.

Also, you can create a shortcut on your phones homescreen to disable pi-hole for x minutes by just a single click (without having to log in.) see: https://discourse.pi-hole.net/t/is-there-an-api-command-to-d...

Another idea might be to remove all blocklists in pi-hole, and only add this list: https://github.com/StevenBlack/hosts

I run pinhole on a Debian Vm on my FreeNas Mini so the perf is not s problem. Could also run fine on any 2008ish era computer you May have faster than the Raspberry Pi if that’s the concern.

Pihole is great! Great revelation was that over 50% of the requests generated by devices in my network are "malicious", ad or tracking related.

The greatest offenders were phone apps and roku TV. Also roku tv scans your HDMI data stream (if you connect your laptop to the Roku TV) which is purely evil!

Also worth to note that to have pi-hole to recognize and block over 50% of the requests, constant gravity updates are required (i have over 3.5M unique domains in block-list)

Also get VPN for your phone. All traffic on my iPhone goes though pi-hole (check OpenVPN and DNS Override iPhone apps in store)

dnscloak is another good ios pihole solution, there are some public piholed +dns-over-https/dnssec dns servers listed one can use if you’re willing to

interesting, i haven't heard about dnscloak. On iPhone i had issues with DNS overrides, since there's no such option in settings. DNS override option only available for app devs which is why i needed and additional app for iPhone DNS settings besides OpenVPN.

yes, dnscloak acts as a local vpn that forwards dns traffic only, i believe. however i also just remembered it was removed from the app store for some technicality recently (apparently vpn apps must have a company as the dev and not an individual? i assume to crack down on shady vpns). hopefully it will be restored soon

Can you explain why what Roku TV does with the HDMI data stream is evil to those of us not in the know?

To quote the privacy policy at https://docs.roku.com/doc/userprivacypolicy/en-us:

"When you use a Roku TV with the Smart TV experience enabled, we also receive information about what you watch via the Roku’s TV’s antenna, and devices connected to your Roku TV, including cable and satellite set top boxes. Using technology such as Automatic Content Recognition (ACR) technology, we receive information such as the programs, commercials, and channels you view, the date, time and duration of the viewing, and how you use the on-screen TV guide. When you use a Roku TV with the Smart TV experience enabled, we use this information to personalize your TV viewing experience and advertisements. Please visit Settings > Privacy > Smart TV experience on your Roku TV to learn how you can enable or disable the Smart TV experience. To understand your choices, see Part IV, Section F below."

I have 2 roku tvs and my live pi-hole blocking log is just a relentless spam from their servers.

Yeah I'm in the same boat even with just one roku

Note that the figures reported may be vastly inflated by poor code on your devices/apps that constantly retries the blocked connections.

how about a 3-pronged approach:

  1) pi-hole
  2) steve blacks hosts file
  3) ublock origin


  3 = most conservative filtering configuration that can easily be tweaked from the browser
  2 = use modules (-e gambling -e porn etc)
  1 = most basic blocking configuration

this way you don't have to do much fidgeting on the router. this comes at a tradeoff for putting some of the burden on the hosts #2 & #3 but with the advantage of better usability for non-tech users.

protip: if you can live with not accessing unicode domains at all (counter measure to avoid domain squatters and some phishermen) patch[1] your dnsmasq and add this in dnsmasq.conf:

   address=/:xn--*:/0.0.0.0

[1] https://github.com/spacedingo/dnsmasq-regexp_2.76.git

EDIT: for mobile I used to have a VPS running openvpn. Make Android connect to the vpn by default routing all traffic through it. Run something like opensnitch[1] to MiTM and whitelist the mobile traffic and sinkhole shit you want to get rid off. This isn't for the fainthearted since new versions of apps might make different API calls and break your rules. Apps will just stop working. If you only have 2 or 3 apps and want to kill traffic from built-in carrier spyware it works nicely and is well worth the effort. Nice way to study what your device does. https://github.com/evilsocket/opensnitch

I agree this is the best setup. Locally using uBlock Origin and Steve Blacks host file is all you really need for a computer but doesn't protect devices which aren't as configurable (eg Roku, gaming systems, phones). By using all three you maximize your protection on your local network and also on your computer when outside that network.

For those that prefer algo, there is an open issue to add Pi-Hole (https://github.com/trailofbits/algo/issues/1258) and in my fork I've added Pi-Hole support: https://github.com/dan-v/algo. It's a really nice setup especially on mobile devices where you typically have to choose between using a VPN or using adblock as they typically require a local VPN hack to function.

Algo user here -- it's really great for ISP-oriented privacy protection. I have a $5/month droplet running Algo in DigitalOcean, and an auto-on mobile config on my iPhone that uses it when I'm not on WiFi.

I also set up my home pfSense router to route all traffic through it (via an IPSec/IKEv2 tunnel). This works great, except when we're using Netflix or Amazon video -- I have to turn it off since those services block DigitalOcean IPs. Still haven't figured out how to configure the pfSense tunnel to route requests to those services through my ISP instead.

This is a great initiative!

I'm using a similar setup. I have my ISP's router in bridge modus, an ERLite‑3 router by Ubiquiti, and a NAS server by Synology. It could've been a router by PC Engines, a Router7, an unRAID, or a Pi too. Ever since I got the Synology NAS (which can run Docker) my Pi is gathering dust.

The NAS runs Docker and uses PiHole(dnsmasq)/Unbound to take the main traffic. The ERLite‑3 router is second choice and uses PiHole/Unbound as well. The setup provides redundancy. All DNS traffic not generated by the router and not coming from the NAS is redirected to the NAS. Both PiHoles have each other as redundant server as well. Both utilize Quad9's DNSSEC servers with DNS over TLS [1]. Though I suppose ideally you want DNSCrypt [2] this should also be possible with Quad9. The advantage this is going to work on any client. Since the ERLite‑3 allows WireGuard (and only that; no OpenVPN or OpenSSH or anything) clients such as my smartphone or laptop have secure and ad-free internet. Since I'm also using Quad9's blocklist, things such as porn are also blocked. The only caveat is that these clients have access to my LAN. While that's intentional, it increases the attack surface of my devices. I shouldn't trust my wife's devices on the LAN remotely. Then again, all internal LAN services are behind strong passwords.

[1] https://www.quad9.net/faq/

[2] https://dnscrypt.info/faq

Do you lose any bandwidth in bridge mode? I’m about buy a new Ubiquiti Edge router and a few APs.

With AT&T I can’t do a true bridge mode or use my own modem. Their gateway has to do the authentication to the network and I’m stuck with it if I want gigabit fiber. People on dslreports lost a good bit of bandwidth on a gigbit connection writing rules on the edge router to authenticate back through the gateway.

Reportedly you can max out gigabit AT&T even if you proxy the authentication bits through the router: https://www.reddit.com/r/openbsd/comments/aaz2rf/openbsd_rou...

Fantastic. It’s only 15 days old too.

> Do you lose any bandwidth in bridge mode?

If you put your ISP's router in bridge mode: no.

If you put the Ubiquiti router in bridge mode: yes (but it allows you to have bridging between WLAN and LAN whilst still on separate ports).

Pihole and Pivpn on Google Cloud Platform free tier is working great for me: https://github.com/rajannpatel/Pi-Hole-PiVPN-on-Google-Compu...

Do you have any privacy concerns about using Google for this kind of service? Serious question.

Author of that guide chiming in! None; Google Cloud Services is not a consumer offering. The monetization approach is different from the model wherein you give up privacy in exchange for a service. More info at: https://cloud.google.com/security/

For those running pfSense: pfBlockerNG is a great package. It allows for IP and DNS based block lists. Between that and the built in OpenVPN server, I have ad (and other malicious traffic) blocking even while remote. Pretty crazy how much it catches on both levels.

The one thing to keep in mind about running a VPN on AWS is that some services just block whole AWS net blocks to stop scrapers.

For instance I used to run a VPN on a free tier micro instance, and I remember not being able to hit Yelp or StackOverflow from a VPN.

Yeah, this is a big problem. I set up a Streisand instance last year just to play around with, and while the set-up was pleasantly simple and the results were great as far as the VPN functionality, it shut down Netflix entirely for me. Definitely a deal-breaker.

interesting! Haven't had any issues w/ netflix yet; wondering if it's an eventually-consistent thing or a static config on their part (i.e. will I eventually get blocked? not sure lol)

From the stuff I was reading/folks I was chatting with when I ran into the problem, it sounds like restarting your instance could sometimes fix it - I assume Netflix is basically playing whack-a-mole with AWS-based scrapers.

Shopping on Amazon generally becomes quite difficult with most VPN providers as well.

Accidently did my taxes with an out of country for active...

This can be fixed with a split tunnel. vpn for adblock dns, your regular isp for tcp traffic.

Then you might as well use 1.1.1.1 and not bother with VPN maintenance.

right. but the vpn allows to have the ad, malware filters on mobile

There is a 1.1.1.1 "VPN" app you know. And with that you don't have to maintain a backend.

1.1.1.1 doesn't block the advert/malware domains, you need something with a blocklist inbetween

Yup, I've had similar issues trying to visit Hulu through a personal VPN in Google cloud.

One of the more annoying things, while traveling is when I forget to turn off my (Linode) VPN and try going to Delta's website https://imgur.com/a/5lllN6J

I just setup the Pi Hole yesterday on the raspberry pi I had sitting on my shelf and I must say it is a god send.

The bonus is my girlfriend noticed the coloring app on iOS she uses doesn't have banner ads and after she's done with a picture, there is no longer popup advertisements.

I only had to make one whitelist so far and it was graph.instagram.com otherwise the app completely doesn't work.

I've noticed tons of blocked requests to graph.instagram.com but no side-effects, why did you need to whitelist it?

Streisand is good for having options to get out of restrictive networks. I would argue AlgoVPN might be a better mix for day to day use.

I have basically the same implementation except it's running on my dedicated openbsd router using openvpn/unbound/pf and scripts. It doesn't support wireguard though but openvpn should be equivalent? I know the wireguard developers claim it's more secure, however I think a properly secured openvpn with certificate based authentication both ways should be fine for a home setup.

For the lazy, someone other than me made scripts to both pi-hole (dns) setup[1] and pf (firewall) setup[2] to block on network level too.

The scripts themselves are very easy to read and vet for bad stuff.

[1] https://geoghegan.ca/unbound-adblock.html

[2] https://geoghegan.ca/pfbadhost.html

The main selling point of wireguard is elimination of insecure ciphers and less chance of shooting yourself in the foot with configuration. the protocol is more lightweight too. openbsd has a port btw:

https://marc.info/?l=openbsd-ports&m=152712417729497

Yeah I know it's supposed to be more efficient too.

But thanks for the link to the openbsd port, I'll definitively look into that! For some reason I thought it was only implemented as a linux kernel module.

I've heard nothing but praise for PiHole setups, though I personally have an AdBlock service on my router.

If you're running OpenWRT, you can install adblock via opkg and there's even a LuCI extension for the web interface.

This blocks the majority of ads on my Roku TV, though YouTube is almost impossible.

What do the failure cases look like with Pi Hole? E.g. broken websites, missing (legitimate) content, apps that crash when they can reach their ad server?

And what kind of workarounds/remediations are available to deal with this, ideally in a user-friendly way for non-technical users?

It really comes down to how many sites you block. It comes with some default blocklists, but wally3k hosts a site tells you the status of different blocklists: https://firebog.net/

As for what is seen when a site fails - well, it could vary. If a JS file fails to load, the site will either look bad or not at all. ANd yeah, some apps may have issues, but pihole comes wiht a built in query log so you can see which devices just made which DNS queries, along with an easy-to-click whitelist button.

There are different blocking modes you can choose from that return different response codes, so you can choose what works best for your environment: https://docs.pi-hole.net/ftldns/blockingmode/

I would recommend checking out the pihole discourse(https://discourse.pi-hole.net/) or subreddit. The community is very active

Thanks for the links, I’ll check those out! Sounds like it’s pretty well thought-out.

I suppose I should make it known that I am a patreon supporter. I may be a superfan but I love what that team is doing

I know it's somewhat irrelevant, but, I would like to note that I've never had this happen. I am running an updated raspberry pi 3 b+ and pi-hole + dnscrypt. It blocks something like 30-50% of my requests daily (top domains are windows and my 2 roku tv's). It allows me to visit sites that detect ublock but also block ads.

It depends on how closely your definition of legitimate matches the block list you’re using, which is why it’s important to choose the right one. My wife and I occasionally disagree about whether this or that link should be blocked but otherwise I’ve had no problem with legit sites working.

The best way around this is to pick the right block lists - I can count on one hand the number of times it's blocked legitimate content (and I run rather a long list of block lists). wally3k's list is good for this - https://firebog.net/ - providing you still stick to the ticked lists, you are unlikely to have a problem.

For working around it when it does happen, the UI is easy to use - go to http://pi.hole, login, and then you can disable it for 10s/30s/5m, or whitelist it if it is something you are going to use a lot.

Thanks, that’s good to know.

When you talk about turning it off briefly, is that for all sites and users?

Yes. It's often pretty easy to look at the query list and just whitelist some obvious domains though, rather than letting everything through temporarily.

I'm running pihole in a Docker container on an Intel NUC, performance is great. Client (browser) performance has also improved remarkably. One problem is getting ipv6 working between host, container and the outside world is non-trivial and non-obvious compared to ipv4 - I still haven't figured it out.

Typically you go into your your router's DHCP settings and populate DNS with the pihole DNS, or disable DHCP on your router and let pihole do DHCP. The gotcha is if you have an ISP supplied router which has no DHCP interface at all: no way to disable it, no way to customize DNS. All of Xfinity's hardware now does this for residential, you have to pay for business service to set DNS servers.

I have a similar problem specifically with DNS and IPv6: my ISP (UPC in Poland) does not allow to change the DNS given for IPv6.

So when a device in my network connects it gets an IPv4 and the pihole DNS for IPv4, but also gets an IPv6 and my ISP's DNS for IPv6.

As I've noticed, all OSes prefer resolving names over IPv6, making the pihole useless.

The only solutions so far are disabling IPv6 which is a shame or setting DNS manually on each device which is impossible on Android.

I had this same problem with Xfinity and just switched to using my own modem & router. Switching to your own hardware should save you $10/month anyway unless you're on some kind of building-wide plan.

For me it was worth it to spend the money and have full control over my hardware.

> The gotcha is if you have an ISP supplied router which has no DHCP interface at all: no way to disable it, no way to customize DNS.

The way I get around this is by putting all of my devices behind another router, and treating the ISP router as if it is part of an external network. It introduces a second layer of NAT (ISP router gives my router a 192.168.0.x address, which it treats as its WAN IP; devices get a 192.168.1.x address), but in practice it's caused me no problems. I've been running a similar setup for close to a decade now.

And IPv6 then becomes an even more confusing clusterfk than it already is.

Looks like we're running Streisand as a VPN in AWS, and pihole at home on a Raspberry Pi. Why not just run PiHole on the AWS instance alongside streisand and save on the added complexity?

I think he actually had pi-hole running on his AWS instance as well. He mentioned using the "connect" button in the AWS console during the pi-hole setup portion.

ah sorry if that's not clear; I started by running Pi Hole on my RBP, then added it to a VPN setup. Nice to have it running at home to block our other devices though

Or conversely, run it all on the pi?

Has anyone got it working with YouTube on Apple TV yet?

Wouldn't you need some sort of intercepting MITM HTTPS proxy to be able to inspect the traffic for ad content? Thought Google was using https and legit domains to drive ads instead of just doing lookups to youtube-ads.google.com etc.

Presumably you'd need a way to inject a cert into the Apple TVs trust store as well, and I'm unclear if that is possible. Perhaps with a developer license?

This is network level, and it's taking the name lookups, which are done before picking the connecting endpoints, so SSL doesn't matter. It's not rewriting the stream, it's just saying, someone is trying to get an endpoint at this particular name and I am going to give it a different IP than the "world" would. Nothing of that is happening inside the https link. Eg, you get https://whatever.com/whateverelse... it will see the DNS lookup for whatever.com regardless of procotol.

I believe rhexs's point is that it's using the same domains for video data and ad data - i.e. you're seeing hxxps://youtube.com/video_data, hxxps://youtube.com/ad_data, hxxps://youtube.com/video_data..., so filtering at the DNS level doesn't work.

The only way I was able to remove ads from youtube on my devices with dedicated apps was to purchase youtube premium.

I use the service so much I don't really mind paying for it.

I would be willing to pay a decent premium to be able to buy a replacement cable modem with this built in. Cool project.

> I even started noticing some network calls our ISP appears to be making

I'm curious what this means. AFAIK the only way your ISP could make network requests that appear in Pi Hole is if they're running some kind of software within your network.

ISP control your router, if you set your DNS on the router to your PiHole, the router might do requests for all kind of reasons.

True for most people, I suppose, but my ISP certainly doesn't control my router.

I think the OP means modem plus router combo that most ISPs offer. The modem is most definitely patched OTA, has been true for most modems since DOCSIS 3.0 was introduced.

Yeah I get it. I always run my own router behind the modem provided by the ISP, if any, so there's no way my ISP would be able to make requests that Pi Hole would see.

Because the ISP is upstream, it could potentially reject all DNS queries from your router or supplant the IP within the DNS query with its own IP.

Sure, but that wouldn't show up in Pi Hole.

With all the people running piholes at home, could a subscription pihole service work? A small vps could run with authentication of some sort (mac-whitelisting seems simple and easy to implement). Allow customization of blocklists per MAC.

That just sounds like AdGuard. Plenty of consumer routers offer VPN server capability for free. That combined with DuckDNS or a similar free service would be a better option I think

I recently set up a RPi with Pi-hole, only to find out that my ISP gave me a router that prevents you from using another DNS server.

I can still use the Pi-hole, but I have to set up the DNS manually for every device. Quite a pain in the ass.

Pi-hole can be used as your dhcp server too, if you can disable the ISP routers DHCP service.

https://discourse.pi-hole.net/t/how-do-i-use-pi-holes-built-...

For a Managed Service of this try https://ba.net/adblockvpn

Disclosure: I work at ba.net

The problem is simple, trust. How can we trust that you're not going to be skimming data going through the VPN, MITM attacks, etc?

There's not even a privacy policy on that page, no indication where your servers are hosted or what laws they are subject to.

For the business solution we setup the servers, and give you the root password (for you to change). We use DO or AWS.

TOS is here https://ba.net/adblockvpn/privacy1.html

Personally I like to use OpenBSD for IPSec VPN, but I like to see guides like this that present a solution for a wider set of problems.

Why not AdGuard DNS?

It has worked well for my home network for months and requires a single one-time configuration (change DNS settings on router).

Then you have to trust them.

I would just like something that blocked Facebook and all its many services. (sign-in, ads, etc)

How user friendly is this for noob, who have no experience in networking?

We can surmise no grandmas are using "Pi Hole" on the daily.

what about grandsons?

Haha, I don't really know but the name cracks me up. You should take a shot at it and the report back any problems you hit so they can improve it for n00bs.

Maybe. I installed on a Ubuntu machine (PC, not raspberry pi). My steps:

1. git clone the pihole repo.

2. run automated install/basic install.

3. configure router's DHCP to assign a static IP to the pihole server, and distribute its IP for DNS.

4. release/renew some other device's IP.

5. browse to a site, then check the pihole console to see if it worked!

Reply on news.ycombinator.com