Block Adware and Malware with /etc/hosts | Hacker News Reader

hosts

github.com

150 pointsfeross78 comments

It's almost the 40th anniversary of RFC 882. Back when the Domain Name System was being invented as the replacement for everyone sharing hosts files, I wonder whether anyone expected that 40 years later there'd be a shared hosts file with its own copyright licence, Docker file, and code of conduct.

Interestingly, from what I remember of the April Fools columns of electronics magazines of the time, one thing that people would not have been surprised by is that the purpose was zapping advertisements.

* https://worldradiohistory.com/UK/Electronics-Today-UK/80s/El...

This is the default list used by the pihole project as well; it's a very good list.

FWIW, something like ublock origin is going to be better for in-browser blocking (for example, DNS poisoning won't work with youtube ads, which are served from youtube.com, but addins like UBO can remove them because they can hook into the actual traffic).

The ultimate adblocking solution is to use UBO in your browsers AND spin up a pihole to block ads on your entire network: https://pi-hole.net/

pi-hole also makes it super easy to block .zip and .mov TLDs.

Don't forget .com which is executable extension.

I'm not worried about the non-techies being tricked into thinking .com is an executable, I doubt they even know its possible. They almost certainly know about .zips

I still don't understand the threat model here. Is there an example of how this could be used as part of an attack?

https://thehackernews.com/2023/05/dont-click-that-zip-file-p...

edit: HN is actually erroring out if I post an example .zip domain of the problem

I'm not doubting you can trick people into engaging with a phishing site that looks like WinRAR. I just don't think the domain makes any difference.

You don't think a url like those in this article [1] makes any difference? You have more faith stopping phishing than I.

[1] https://medium.com/@bobbyrsec/the-dangers-of-googles-zip-tld...

If I hover over the link like my phishing training says to do, Chrome correctly shows it as "https://v1271.zip"

If I don't bother hovering over the link then the domain never mattered at all. Could have hosted the file anywhere since the link text in HTML (including HTML email) need not have any relation to the destination. You can simply write a legitimate github.com link but the href goes somewhere else.

Any situation where you're relying on users to visually inspect every link and decide if the domain looks plausible is already a security failure.

If I'm downloading and executing files that are randomly in a Medium article or in my inbox from an untrusted source then we're in trouble whether they are hosted on github or not.

> If I hover over the link like my phishing training says to do

How do you hover on mobile? Instead of training my family members to hover on their phone's I will continue to just block these TLDs.

I’m confused what you’re advocating. If your family isn’t checking the domain before downloading and executing files then the .zip tld doesn’t matter. They’re just as vulnerable to phishing without it.

> I’m confused what you’re advocating.

I'm advocating using a pi-hole to block problematic domains on your network using list like the one from StevenBlack and entire TLDs that could be confusing. If you don't think .zip & .mov could be confusing then feel free to not block them.

The explanation I've seen is that certain programs (mail, chat, forums, etc) automatically recognise urls that are not hyperlinked and convert them into a hyperlink.

E.g. "example.com" becomes "<a href="http://example.com/">example.com</a>".

Such features would now (potentially) also work on mentions of zip files, i.e. on messages that state "download install.zip from github".

Edit: ghi, HN actually does this. Not yet with zip TLDs though.

As I understand, the attack is preformed by when someone mentions attaching a file to an email (i.e. "I've attached invoice.zip"), then that filename becomes a clickable link, and the user clicks it expecting to download the file, but is actually taken to the attacker's website which looks like WinRAR, and download malware from there.

To me this attack sounds too convoluted to actually get people, since you'll need to buy a lot of domains to match someone's email's contents

attachment.zip ?

<a href="https://malware.example.com/">attachment.zip</a>

Which email client automatically creates links? Virtually everyone uses HTML email, which can link any text to any destination. If your users are clicking links to download files that they got via email from untrusted sources...then the domain doesn't really matter.

I just tried emailing myself a few links and they all got automatically hyperlinked by Outlook on Android. Here are the links I tried

testlink.com link.gov abc.zip test.xyz

I agree about being able to hyperlink pretty much anything having the same effect, that's why I don't think the .zip TLD is a big issue

To be fair all extensions are potentially executable. Extensions are just a mapping to an application maintained by the OS. .com is just mapped to a program that executes by default. File type is a whole different ball of yarn to untangle.

Pi-Hole and PiVPN [0] is a great setup for DNS filtering on-the-go. The setup for PiVPN makes it foolproof as well.

When my Pi stopped working and I couldn’t find a new one, I gave up and moved to NextDNS [1] and have been very happy.

[0] https://www.pivpn.io/ [1] https://nextdns.io/

Extra fun: measure bandwidth use before and after implementing, the savings are fucking nuts. All that adtech webshit loading in hundreds of mb of obfuscated JS and assets to sell you widgets…

To ensure there are no nefarious redirects in such a massive list, this one-liner comes in handy[1]:

    grep -v '#' steves_hosts | awk '{print $1}'| sort | uniq -c

[1] https://news.ycombinator.com/item?id=11456562

What is a nefarious redirect? Does it look for entries without 127.0.0.1?

A redirect to some unexpected IP address. Here's what the output looks like:

    190625 0.0.0.0
      3 127.0.0.1
      1 255.255.255.255
      3 ::1
      1 fe80::1%lo0
      2 ff00::0
      1 ff02::1
      1 ff02::2
      1 ff02::3

There's a concern that a malicious list could point domain names to a malicious IP address. I don't think its a big concern with https:// since the cert will be invalid, but it's still a concern.

I suspect it’s very likely that somewhere in the world is a domain-validation server, used by a trusted CA, which has this very anti-advertising hosts file installed onto it.

Why? A CA would be unable to issue certificates for advertising sites with that configuration.

It could be a lesser-known CA, perhaps the national CA of a small country (under 2m people) that normally only issues less than tens-of-thousands of certs and only exists for regulatory reasons (e.g. the country requires all of its own gov services to use its internal CA, while all commercial/popular services use CA based in another country, usually LetsEncrypt)

Someone explicitly called out and removed the Useless Use of cat but yet left the Useless Uses of grep, sort, and uniq in, I see. (-:

* https://porkmail.org/era/unix/award#grep

    awk '!/#/ { seen[$1]++; } END { for (k in seen) { print seen[k],k; } }' steves_hosts

I'd rather have the readability of the first command than the "efficiency" of your example.

Guilty as charged!

I love awk as much as the next person and I've written some pretty large scripts with it, but the original grep/sort/uniq is a lot easier to understand in two seconds. I don't think it's "useless" at all.

Also your awk isn't exactly identical as it's not sorted, but in this case that probably doesn't matter.

Same, I think I use grep | sort | uniq almost on the daily.

One of the advantage of the array[item]++ / for (item in array) construct is that it avoids possibly expensive sorts (at the cost of generating potentially large arrays), and tends to be quite fast.

That's even if you end up sorting the output by frequency, as the summary listing is typically shorter than the overall input data:

  {a[$1]++}

  END {for (i in array) printf("%6i  %s\n", a[i], i) | "sort -k1nr | cat -n";}

The accumulator/loop idiom is ... fairly readily recognisable to someone familiar with awk.

    find . -name hosts -exec grep -v '^#' {} \; | awk '{print $1}'| sort | uniq -c
    43029
     204 #
    4289934 0.0.0.0
    10763 127.0.0.1
      18 255.255.255.255
      54 ::1
       1 analytics.shein.co.uk
       1 analytics.shein.com
       1 auxilium.ftb.team
       1 bstats.org
       1 fe00::0
      17 fe80::1%lo0
      33 ff00::0
      17 ff02::1
      17 ff02::2
      17 ff02::3
       1 mcmc.dev
       1 mcstats.org
       1 metrics.shmeeb.net
       1 openeye.openmods.info

And looking at those last ones, looks like some files have a different format. For example, the data/minecraft-hosts/hosts file,

     ~/D/P/hosts   master   grep -irn 'openeye.openmods.info' .
    ./alternates/fakenews/hosts:174307:0.0.0.0 openeye.openmods.info
    ...
    ./hosts:174306:0.0.0.0 openeye.openmods.info
    ./data/minecraft-hosts/hosts:9:openeye.openmods.info
     ~/D/P/hosts   master   grep -irn 'metrics.shmeeb.net' .
    ...
    ./data/minecraft-hosts/hosts:8:metrics.shmeeb.net

I used to use this hosts file some years ago: https://someonewhocares.org/hosts/

Forgot about it for a few years, but this post jogged my memory.

We currently use a PiHole for house-wide, network-wide ad and telemetry blocking, though, but perhaps that hosts list is useful to someone else.

I've been using this list locally and on my PiHole for years. Love it!

Does anyone have any updates on Mike Burgess (from winhelp2002.mvps.org)? He used to keep my go-to hosts file, but his health seems to have declined in the last few years. I would like to know if he is still around and if there is currently any method to donate or support him. Any news would be greatly appreciated.

Reminds me of SlashDot where there was a relatively well known submitter that was a huge proponent of host-based ad blocking on Windows and was also a proponent of getting very angry at almost anyone else.

(Edit: typo)

That’s the most believable thing I’ve read on the Internet for ages.

Slashdot has a, shall we say _engaged_, community.

heh I made the mistake of engaging with that once. oh boy....

They could not wrap their head around the idea that the hosts method is limited in some use cases.

This could not be blocked without blocking the main host too. https://example.com/adverts-folder/advert.jpg

Now it sort of works in practice as most of the advert networks do not host on the same domain.

It is good at catching out whole bad domains but partial ones it is not very good at. Which some domains will do using a reverse proxy.

Also at one point a hosts file was linear scan. No sort of hashing or binary search lookup. Not sure if that was ever fixed in windows or linux. So a small number of hosts it was ok. But as the file grows it starts to add up.

I'm half expecting a torrent of abuse in reply to you now from whoever that was

oh no...

Plus you can't forget Natalie Portman and Hot grits down your pants.

What a strange place that was.

Works good for blocking fake news and sites you don’t want your elderly parents getting tangled with.

Those licenses are a little silly. You can't copyright a phonebook.

https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....

Related to this, my best cure against procrastination has been adding some distracing websites to /etc/hosts and pointing them to localhost.

I've noticed a speed increase by running these tools along with the hosts file: http://www.analogx.com/contents/download/Network/qdns/Freewa... http://www.analogx.com/contents/download/network/fc/freeware...

And https://mayakron.altervista.org/support/acrylic/Home.htm works as well, and it also accepts wildcards.

Hey this is my repo!

I'm happy to answer any questions arising.

I must say: the lists we offer — 31 variants in all — result from fine work by all our curators. Some are extremely diligent and maintain their lists every day. It's just remarkable what dedicated people can do together over time.

On the Mac the new Little Snitch Mini is really nice for this:

https://www.obdev.at/products/littlesnitch-mini/index.html

Sure, you could manage /etc/hosts manually, but Little Snitch Mini has a nice interface for managing blocklists, auto-updates them, and has nice visualization, and you can also manage blocks per app.

Portmaster is the best Windows alternative I've found.

Much better to do so at router level via pihole or Adguard Home. That way it catches the calling home from all the various electronics too

Yes. pihole coupled with a router that can intercept outbound traffic on port 53 and force it to the pihole is how my home network is configured. That way it catches all those sneaky devices that have hard coded dns servers.

I've also added (pfSense with pfBlocker) a list-rule to block all Public DNS servers (https://public-dns.info/). If you want to resolve a name, you gotta go through my DNS server.

The UDP:53 block gets almost everything, but I'm preparing for DoH.

The only problem with this is that on interface up/down it can take a few minutes for the network to be available. This may not be an issue with a server or idle desktop but may be confusing for non tech people or laptop users.

If you can manage it, a pihole/pivpn instance may be better. Both of those can be installed on any debian distro easily.

https://github.com/Ultimate-Hosts-Blacklist/Ultimate.Hosts.B...

Don't forget about hosts.deny (but you probably need it at your router if you are behind one).

Shell scripts that download more shell scripts. Kind of terrifying, no? I feel like I would personally rather take my chances getting scammed out of a CS skin than running these installers as root at will.

Download the raw file.

intiially i wanted to point out a potential issue with using large /etc/hosts file in windows, but ended up finding a (potential) fix in this thread.

regardless, after using this list for a long time, i do recommend setting up your own dns server or use a service that does (such as pi-hole and nextdns already suggested by others) with this list enabled.

on mobile, i found that ad-blocking dns services that offer DNS-over-HTTPS (such as adguard dns) offer a much more elegant option that does not require an app nor configuring every network you connect to.

I’m a fan of dbab. It’s DNS based as well but pretty light weight using dnsmasq in the backend. No fancy GUI and such like pi-hole, just simple domain blocking.

hosts file will grow with records + you need to run specific scripts to update it. One of the similar options will be to use DNS resolver with built in filters (like pi-hole, but "cloud"), for example: https://nextdns.io, even free version is OK.

Didn’t work well for me last time I tried, I think entire IPv4 stopped for ~5min on each reboot with this.

To resolve in Windows 10[1]: Disable DNS Client by changing the value data for HKLM\SYSTEM\ControlSet001\Services\Dnscache\Start from 2 (automatic) to 4 (disabled) (recent versions of Windows 10 no longer allow disabling DNS Client via services.msc).

[1] https://tinyapps.org/blog/201809300700_large_hosts_file.html

Not gonna help if the malware authors decide to embed their own implementation of DNS.

I think the idea is to prevent downloading malware in the first place, not to hamper it while it's running.

This isn't a functional approach in the age of DNS over HTTPS (DoH).

0.0.0.0 connects to your own machine. You should not "block" domains by assigning them to 0.0.0.0. You should instead return NXDOMAIN.

There's no host file syntax for NXDOMAIN, it has to be an IP. 127.0.0.1 with Port 80 assigned to a dummy page can be an alternative solution if you don't like 0.0.0.0.

The hosts file is not the right tool to do this.

Preach.

99% of people don't understand (because they don't even try) what you can run a full resolver even on Windows machines. butmuhpihole

Ap akmbgusnnine

A tech as old as Kazaa lite. But people like to use heavy ram chrome browser extension to do the same thing.

It's not the same thing, browser extensions can block at a much more granular level than entire hostnames.

Er, is uBo a RAM hog? Glancing at mine right now it's consuming ~36 MB, i.e. about the same as a single small tab.

I use Brave (not so much for the built-in ad block, but mostly because it's not actually Chrome) and I've noticed that more and more sites have ad-block detection that appears to work by sending a unique token to their ad server and then not loading content until after the token has been verified to have been received. I imagine there are ways around this on a per-site basis, it's generally easier to just give in and disable the ad blocker.

NoScript usually bypasses this behavior. Especially when the content is delivered but hidden with JS DOM manipulation.

I wish NS worked on Android as a generic domain blocker

Reply on news.ycombinator.com