Hacker News Reader

Comment by Animats | Hacker News Reader

If you block those internal subdomains from search with robots.txt, does Google still whine?

I’ve heard anecdotes of people using an entirely internal domain like “plex.example.com” even if it’s never exposed to the public internet, google might flag it as impersonating plex. Google will sometimes block it based only on name, if they think the name is impersonating another service.

Its unclear exactly what conditions cause a site to get blocked by safe browsing. My nextcloud.something.tld domain has never been flagged, but I’ve seen support threads of other people having issues and the domain name is the best guess.

I'm almost positive GMail scanning messages is one cause. My domain got put on the list for a URL that would have been unknowable to anyone but GMail and my sister who I invited to a shared Immich album. It was a URL like this that got emailed directly to 1 person:

https://photos.example.com/albums/xxxxxxxx-xxxx-xxxx-xxxx-xx...

Then suddenly the domain is banned even though there was never a way to discover that URL besides GMail scanning messages. In my case, the server is public so my siblings can access it, but there's nothing stopping Google from banning domains for internal sites that show up in emails they wrongly classify as phishing.

Think of how Google and Microsoft destroyed self hosted email with their spam filters. Now imagine that happening to all self hosted services via abuse of the safe browsing block lists.

if it was just the domain, remember that there is a Cert Transparency log for all TLS certs issued nowadays by valid CAs, which is probably what Google is also using to discover new active domains

It doesn’t seem like email scanning is necessary to explain this. It appears that simply having a “bad” subdomain can trigger this. Obviously this heuristic isn’t working well, but you can see the naive logic of it: anything with the subdomain “apple” might be trying to impersonate Apple, so let’s flag it. This has happened to me on internal domains on my home network that I've exposed to no one. This also has been reported at the jellyfin project: https://github.com/jellyfin/jellyfin-web/issues/4076

In my case though, the Google Search Console explicitly listed the exact URL for a newly created shared folder as the cause.

https://photos.example.com/albums/xxxxxxxx-xxxx-xxxx-xxxx-xx...

That's not going to be gleaned from a CT log or guessed randomly. The URL was only transmitted once to one person via e-mail. The sending was done via MXRoute and the recipient was using GMail (legacy Workspace).

The only possible way for Google to have gotten that URL to start the process would have been by scanning the recipient's e-mail.

Not quite. Presumably the recipient clicked the link, at which point their browser knows it and, depending on browser and settings, may submit it to Google to check if it's "safe": https://support.google.com/chrome/answer/9890866#zippy=%2Cen...

Good point. Thank you.

I've read almost everything linked in this post and on Reddit and, with what you pointed out considered, I'd say the most likely thing that got my domain flagged is having a redirect to a default styled login page.

The thing that really frustrates me if that's the case is that it has a large impact on non-customized self-hosted services and Google makes no effort to avoid the false positives. Something as simple as guidance for self-hosted apps to have a custom login screen to differentiate from each other would make a huge difference.

Of course, it's beneficial to Google if they can make self-hosting as difficult as possible, so there's no incentive to fix things like this.

Well, that's potentially horrifying. I would love for someone to attempt this in as controlled of a manner as possible. I would assume it's possible for anyone using Google DNS servers to also trigger some type of metadata inspection resulting in this type of situation as well.

Also - when you say banned, you're speaking of the "red screen of death" right? Not a broader ban from the domain using Google Workplace services, yeah?

> Also - when you say banned, you're speaking of the "red screen of death" right?

Yes.

> I would love for someone to attempt this in as controlled of a manner as possible.

I'm pretty confident they scanned a URL in GMail to trigger the blocking of my domain. If they've done something as stupid as tying GMail phishing detection heuristics into the safe browsing block list, you might be able to generate a bunch of phishy looking emails with direct links to someone's login page to trigger the "red screen of death".

This reminds me of another post where a scammer sent a gmail message containing https://site.google.com/xxx link to trick users into click, but gmail didn't detect the risk.

Chrome sends visited urls to Google (ymmv depending on settings and consents you have given)

Yes, my family Immich instance is blocked from indexing both via headers and robots.txt, yet it's still flagged by Google as dangerous.

I'm kind of curious, do you have your own domain for immich or is this part of a malware-flagged subdomain issue? It's kind of wild to me that Google would flag all instances of a particular piece of self-hosted software as malicious.

G would flag _some_ instances.

Possible scenario:

- A self-hosted project has a demo instance with a default login page (demo.immich.app, demo.jellyfin.org, demo1.nextcloud.com) that is classified as "primary" by google's algorithms

- Any self-hosted instance with the same login page (branding, title, logo, meta html) becomes a candidate for deceptive/phishing by their algorithm. And immich.cloud has a lot of preview envs falling in that category.

BUT in Immich case its _demo_ login page has its own big banner, so it is already quite different from others. Maybe there's no "original" at all. The algorithm/AI just got lost among thousands of identically looking login pages and now considers every other instance as deceptive...

I have my own domain, and Immich is hosted on an "immich" subdomain.

I see, thank you for clarifying.

I'm guessing Google's phishing analysis must be going off the rails seeing all of these login prompts saying "immich" when there's an actual immich cloud product online.

If I were tasked with automatically finding phishing pages, I too would struggle to find a solution to differentiate open-source, self-hosted software from phishing pages.

I find it curious that this is happening to Immich so often while none of my own self-hosted services have ever had this problem, though. Maybe this is why so many self-hosted tools have you configure a name/descriptor/title/whatever for your instance, so they can say "log in to <my amazing photo site>" rather than "log in to Product"? Not that Immich doesn't offer such a setting.

Reply on news.ycombinator.com