Before you buy a domain name, first check to see if it's haunted
bryanbraun.com
bryanbraun.com
I purchased a valuable premium domain to host a personal art collection (of anime cels). For some bizarre reason, the site was inaccessible from my work computer and it was de-listed from Google even if I typed the url itself into search.
I hired a square space specialist to figure out why, to no avail. I then begged our company’s CISO to investigate and it turns out we had some firewall setting on UniFi that blocked the domain because it appeared on a list. Once I checked way back, it turns out that it was as an anime porn aggregator years back. I personally reached out to all the web filters out there (Google, Symantec, bing) and one by one filed tickets for them to mark it as art instead of pornography and it worked. I am now properly crawled on Google but still MIA on Bing, search console is giving me some BS error that’s incomprehensible, typical of MSFT.
I have a +100 cel backlog that I need to catalog and photograph. Was planning to do it this holiday season so check back in.
We're talking like 20 years back. Holy shit, my brain is getting jostled by this sudden tsunami of forgotten memories.
EDIT: Digging around on Wayback Machine (obviously NSFW, for the curious), apparently it was actually still around until somewhere between 2018 and '19 when it finally died. The snapshots from around 2007 are peak Web 1.5 design with stuff like affiliate buttons and table layouts. Man I miss that era.
Also the resources->galleries was useful, found some new but actually old sites to check out.
Yahoo Auctions is more popular over there and proxy services (I use Buyee) make it pretty simple bid/buy and not too much more expensive if you wait for their (Buyee) coupons.
I had no idea such a thing existed.
If you can set up your own domain why would you need someone that specializes in a super limited non technical frontend for customizing prebuilt web templates?
It's not that the smooth path you can get via nepotism is the base way things work which people who don't "know a guy" are excluded from. Rather, everything is falling apart and shitty, and if you're lucky, you occasionally get to circumvent that shittyness.
Well, obviously it isn't if you're not in the 1%. If you're in the 1% then that's the way the world has always worked and you don't know anything differently.
I don't believe that human society can, practically, get particularly close to the ideal. I question the choice of fatty meat as a substrate for minds.
For my money, I'd suggest that merit will get you further today than in the days of letters of recommendation, but that failures of meritocracy are more visible.
Where I am there is no forced disclosure, no costs costs assigned, and it is $150 to file.
And while a lawyer can represent a large firm, an employee has to be present, and the lawyer cannot use excessive legalise, the court is carried on in plain language... with the judge expaining things to you if you don't userstand.
That's pretty accessible.
The biggest risk is not knowing about no required discovery, and costs. Lawyers for big corp will ask for things, and hope you work your tail off. I just say no.
They will also elude to how expensive this will be, to which I typically snort.
Said large companies typically spend 50k to 100k on lawyers, and I spend $150 and a dozen or two hours of my personal time.
All very amusing.
Anyhow, a good equalizer.
Sadly, I think this would be instantly gamed by abusers. They would release the domain name and attempt to register as a new owner or start repeatedly doing handoffs. It's difficult to tell who the owner is changing between and whether or not the new one is a better actor than the former.
This doesn't seem like that hard of a problem to solve, because these are domains with negative reputation, i.e. worse than zero.
So if a) the domain is no longer hosting any of the stuff previously complained about and b) is no longer receiving new complaints over a period of a year, it costs you nothing to reset the domain to zero. Because the bad actors don't have to behave for a year to get back to zero, they can just register a new domain.
All you're doing is giving the new owner the same fresh start that anybody can get by buying a never before registered domain for the same price as a year's renewal on the existing one.
A google rank at zero and lots of 2 hop routes to your site that google can either penalize for being an accurate historical record or not is better than a rank of zero and a domain that has never been in historical artifacts.
If they were going to consistently use the same domain for links while they churn through hundreds/thousands a year for Google, the extra cost for one extra renewal for the persistent domain would be entirely negligible. And on top of that would make it trivial for Reddit/Facebook/etc. to disable all the historical links because they all go to the same scam site.
- Any empty domain starts with the same reputation
- Registering a new domain is a 0 cost action
- The eng effort to reset domain reputation is 0
Certain domains are used by abusers more often, usually due to them being cheaper. Forcing them to move domains is extra friction to the abusers which "haunted" domains force more than the proposed new system.
For the last point, I think it's simplifying a complex system change. Even if the new system was marginally better, it could be a large eng effort and not worth pursuing.
edit: styling
What basis would you have to do otherwise, and if there is something (like TLD), why wouldn't "resetting to zero" in terms of past content just mean resetting to that zero?
> Registering a new domain is a 0 cost action
No, that registering a new domain has a similar cost to renewing an existing domain, which is a valid assumption. In fact, the new domains are often cheaper because registrars often discount the initial registration as a loss leader with the expectation that people will make future renewals at a higher price.
> The eng effort to reset domain reputation is 0
It is the job of the party operating that system to make it operate as correctly as feasible. Needlessly causing collateral damage purely out of laziness and unaccountability is how you get people showing up at government offices demanding for you to be regulated or broken up, if not showing up at your offices with a disposition to cause bodily harm.
> Certain domains are used by abusers more often, usually due to them being cheaper.
Running out of domain names is physically impossible. There are more possible domain names in any given TLD than there are atoms in the observable universe. So the low price is going to be the price set by the registry for that TLD.
Whether the TLD itself has some reputation is orthogonal to the reputation of one domain in that TLD relative to another one in the same TLD. Moreover, you would presumably do the same thing for the TLD -- if one TLD is doing promotion and has $1 registrations this year and then gets used for a lot of scams, and then next year it costs $15 and so do the renewals so the scammers move to a different TLD, the reputation of the TLD should be reset just the same as the individual domains.
> Even if the new system was marginally better, it could be a large eng effort and not worth pursuing.
If the primary goal is to reduce engineering effort then the obvious solution is to delete the entire reputation system so it doesn't have to be maintained anymore. If the primary goal is to make it work well then you have to, well, you know.
Fair enough, but I'm not sure it resolves "haunted" domains as a TLD which is often abused could have a lower "0" reputation and thus by default is "haunted". Perhaps it lessens the impact though by how much is quite opaque to us.
> Whether the TLD itself has some reputation is orthogonal to the reputation of one domain in that TLD relative to another one in the same TLD.
I think this depends on how reputation works and is not so clear. Registrars for these TLD also have a responsibility but have no incentives to stop abusers. If TLD domain reputation is not orthogonal to reputation individual domains on that TLD then that could be an incentive for them to also crack down on abuse as their domains have bad SEO etc.
> If the primary goal is to reduce engineering effort then the obvious solution is to delete the entire reputation system so it doesn't have to be maintained anymore. If the primary goal is to make it work well then you have to, well, you know.
I think this is the most uncharitable interpretation. The eng effort could go to features that improves other customer experiences affecting more people.
A good technique to evaluate ideas though is to try and view it from different perspectives.
In this case from the owner of a non-haunted domain. Can you see any potential problem with your idea when viewed from that perspective?
Now, if there are potential problems, consider the relative sizes of the two groups. Do the benefits to one outweigh harm to the other?
This technique can be used every day with pretty much any idea.
I would want to experiment judging them based on what they’ve been seen to do in the past month.
If you remove the blacklist, they’d just stop doing that and it would be even easier for them.
Yes, the spammers can sit on their domains once blacklisted, renew them, and redeploy their spam on them 12 months later, but they'd have nothing to gain from the reuse, since the names of their domains are just nonsense anyway.
I’m guessing that would complicate blacklist maintenance quite a bit, which is why we aren’t seeing it work that way.
Most of these blacklists (at least initially) were emergency type measures - ‘block these spammers’, then move on with life.
Blacklist maintainers would need to maintain date first seen/date last seen info, and purge/re-add correctly.
Technically, seems like an ‘append only’ type thing is what they’ve been doing for the most part.
As this evolves and the idea that these do need some kind of expiration or we end up with more maintenance headaches becomes more widely known, maybe eh?
Or if there is some kind of legal rules around it.
I'm not up to date with SEO so unsure whether Google would (or is able to) reset the domain's backlink profile, I'd guess it would be possible. A lot of the value of using expired domains is for backlinks (or at least was)
So I checked the Bing Webmaster Tools. URL Inspection says "Discovered but not crawled - The inspected URL is known to Bing but has some issues which are preventing indexation. We recommend you to follow Bing Webmaster Guidelines to increase your chances of indexation."
That's quite unhelpful. What's more, when I open the "Live URL" tab, it says, in green: "URL can be indexed by Bing."
It's a simple static Hugo site hosted on Cloudflare R2 (DNS mapped directly to bucket). https://pagespeed.web.dev gives it a score of 100 in every category.
Anyone else had something like this happen?
It's a handwritten HTML website, enhanced with JS but not reliant on it, hosted on Cloudflare. Not quite a 100 in every PageSpeed category, but just about.
I've seen a few sites become de-indexed and the 'give away' is the type of results that first appear when the penalty is eventually lifted. For example, just a dozen or so urls with really weird query strings that never existed before. The real stuff does come back after time though and, in my limited experience, it's a one-off incident.
Just to add, not many sites are insignificant enough not to attract negative seo - especially this type of low-level, zero cost malarkey.
HSTS (which forces browsers to validate HTTPS when connecting) asks browsers to cache the configuration for a set "max-age". Some sites set huge values here, like Twitter's 20 year max-age[1]. There's also the preload lists [2] to consider. This creates a problem if you want to serve non-HTTPS/unencrypted HTTP on your new domain and the previous owner didn't.
MTA-STS [3] is another variant that's becoming more popular. It limits which mail servers your domain uses and enforces TLS certificate verification. "max_age" is capped to a year by the RFC. If you don't set your own policy, then the previous domain owners policy would impact any senders who previously cached the policy.
Thankfully HPKP (key pinning) is obsolete, otherwise you'd also need to worry about old pinned keys too. That RFC recommended, but did not enforce, a 60 day max-age limit.
These are especially tricky as the old security policy only lives in the caches of any end-user devices that previously connected to the domain. Double haunted.
[1] https://alexsci.com/blog/hsts-adoption/
[3] https://alexsci.com/blog/smtp-downgrade-attacks-and-mta-sts/
So the sender is supposed to obey the normal DNS TTL caching period, and re-query the assertion record if TTL expired. It should re-fetch the MTA-STS policy if the 'id' value in the DNS assertion changed, or the max_age in the previously fetched policy has expired.
> RFC 8461 section 3.3: Conversely, if no "live" policy can be [...] fetched via HTTPS, but a valid (non-expired) policy exists in the sender's cache, the sender MUST apply that cached policy.
You'll also need to host a "none" policy doc. Full instructions are here: https://www.rfc-editor.org/rfc/rfc8461.html#section-8.3
Look at the milka.fr problems... Milka is also a female name over here, and that already proved to be a problem in france. But so are Mirka and Minka so yeah... no domain for them? Also Micka. Oh and mivka is (beach) sand. Want to sell beach sand? It's just one letter away from milka, so no domain for you either.
Still seems better to raise the issue as early as possible so they can find a solution (appeal or chose a different domain) before investing into the unusable domain name. It would also mean that the dispute is at a layer (ICANN) where you at least theoretically have some rights instead of at the hands of a megacorporation that thinks the best way to reduce customer support costs is to make it impossible to get support.
Just one more place where the web gets screwed by a company too big to have to do basic customer service.
- knowing all the complexities of every local, state, federal, international jurisdiction that might interfere with the whitelist
- awareness of the content in question which could be millions of subpages
- a customer support team that is definitely not incentivized based on tickets triaged per day, but is somehow incentivized to spend hours on “whale” tickets.
- going through ticket history and solving the problem for everyone now that its policy to solve this
- dealing with the inevitable rush of fraud that follows every tiny change in google systems
Some practical advice here: do not change your canonical domain[1] name unless you really really have to.
If he had just set his fun new domain to redirect to the existing domain, instead of making the new domain the canonical, it likely would have had no negative effect.
I’m not saying this is how things should work. But the practical reality is that your domain name is like a Social Security number: it’s the basis for assigning a type of reputation score, even though it was not intended to do that originally.
[1] The domain at which your web pages finally load, after all redirects have completed.
I don't think it's possible to fix this problem without also helping bad actors. Maybe it's a problem that just isn't worth fixing. Just don't buy preexisting domains unless it's a project big enough to justify the necessary cost of due diligence.
There is a finite amount of short, memorisable names.
If you've ever gone to a nightclub or bar which has no name, only its street address number, that's what has happened there.
Checking web archive is a basic operation to test if site was hosting anything fishy - not only pirated stuff or porn - often websites has been hacked and changed into link farms or simply were bought on aftermarket simply to use it's SEO value to pass the strength to other domains.
Anyways good point regarding email filters.
I think the mistake here is the redirect old to new. That is always risky so only do it if deseprate. In this case I would have done the redirect from new to old. Then just use the new as a vanity url.
I have never hear of anyone being denied business because their car has a bad reputation from a previous owner.
I set up a catch-all for personal use and wasn't expecting to get flooded with emails.
I was getting business emails, people trying to send money by Zelle, etc.
I was kind of hoping to get something good that I could take action on in the market, so I left it on for a little bit, but then I felt bad that people's emails were not getting answered (at least bouncing), so I turned off the catch-all. Oh well.
Even automated queries are likely to spill the beans. Someone else could snag the purchase before you, or bid up the price. But it's a risk you may need to calculate.
Wayback machine would've saved me there, had I done my due diligence!
Here in the UK with EE/BT that correctly redirects to automattic.com, but it might not for you depending on your ISP.
The wayback machine shows adult content links prior to the domain being put on sale, hence the blocking.
But it does require manually reporting false positives to each vendor
Using dig:
$>dig yourdomain.tld
1.2.3.4
$>dig -x 1.2.3.4
evilcorp.com
I am admittedly a bit distant from SEO. The above is not true and hasn't been true for a long time.
Managed to get a takedown notice thanks to that idiotic "feature" while not even aware the domain is serving anything
That doesn’t sound like old info - that sounds like someone might still be reporting it for abuse even after the domain changed owners.
It’s like when cars took over the streets, and instead of blaming cars for being dangerous for regular people using the streets for walking, the concept of “jaywalking” was invented by car companies to place the blame on people for daring to obstruct cars. Or the concept of “personal carbon footprint”, commonly used to move blame from companies to individuals, when in reality whatever individuals, even in aggregate, could do is utterly insignificant compared to what companies and legislation could accomplish.
These kinds of blacklists exist because these domains have been used to host scams or distribute spam (or some other malicious activity) in the past. They're there to protect people (e.g. so that Firefox can disply a "warning: this site is a scam") and reduce abuse. They're not just there so people at Google can get a good kick out of blacklisting random domains.
With tricks like this, it's not a surprise to see why the companies operating blocklists are hesitant to make this process easy; after all, what's to prevent the phishers from temporarily stating that the issue has been resolved to get out of the denylist, and then restarting their campaign again?
Still, an improvement over what they were previously using I guess?
The issue is with the issue: people/systems (big and small) blacklisting an ownable identifier pointing to some ownable content without any care for the lifecycle of either.
Painting this with a social brush is extremely unhelpful and is guaranteed to derail conversations for no benefit whatsoever.
Does the lifecycle matter much, though?
Kind of like a carfax report. Tells you whether a vehicle you’re buying has been in an accident before (if it has, the value goes down because maybe there’s some latent issue that isn’t obvious at the time of purchase)
It would be nice if ICANN had some equivalent of a carfax for domains, perhaps even with a requirement that registrars expose at time of purchase whether a domain has been misused in the past (and who the prior owners were, or at the very minimum what the historical DNS records were).
Basically you want to avoid buying a “lemon” domain by accident.
I place zero fault/blame on “powerful entities” maintaining lists of domains used for spam/scams. How else will we protect grandma?
"Heads up, this is a pre-owned domain. Do you want to get the Namefax for $0.99 before you buy?"
On the other hand, a domain reputation at Google et al. is more like Carfax reporting “This car was once parked at the same street where a horrific mass murder took place.” If this was a problem since, let’s assume for the sake of argument, the police would pull you over all the time if you drove it, it would still not be a problem with the actual car; the problem would be the police, and fixing police behavior would be the only workable solution. Using Carfax as an analogy still places the blame on the domain owner, not on Google et al.
This kind of issue is inherent to any system where identifiers are recycled, particularly when that recycling happens on demand. It's not "fixable", at best it's combatable. And trying to language police away the symptom and blaming it all on the pivotal participants supports and achieves neither.
If it was a reputation problem where, say, end clients with web browsers would each have a separate and uniquely derived negative opinion about domain names, this would indeed be a “bad reputation” problem and not a Google problem, since the problem could not be fixed at the Google side. But with domain reputation being so centralized, the problem is at the center.
How could it not? It's essentially the same issue as an unmaintained phonebook or a map. What's at a given address or phone number changes, and if your solution is not equipped to handle that change, your solution is bad.
But that’s not a fixable problem in my eyes. At least not without extreme and sweeping changes driven by some kind of government regulation or ICANN mandates which, if enacted, would probably be highly criticized on HN.
There are just too many block lists for domains (literally thousands if you include open source ad blockers).
The lifecycle “should” matter in a perfect world, I agree.
As you say there are plenty of volunteer maintained blocklists as well, and there are also the countless privately deployed filters using those lists, which may or may not get updated properly. That's the "little man" part, and is why I think the characterization the thread starter was trying to push is ill-fitting.
In a perfect world, when your legitimately good content isn’t being surfaced by Google, it’s a failure on their part, and their problem to solve, not yours. In practice, it is your problem and you have to do a bunch of work to help them see that their current assessment of your domain name is no longer accurate.
You're right, the fault lies with the search engines, but in practice it sure feels like the domain itself is tainted somehow.
Something terrible happened here in the past.
The intangible spirts from this terrible event remain.
The new owner discovers his pictures scream at him and his closet constantly fills up with blood.
The fault, ultimately, belongs with the one who did the terrible deed.
With some domains, you merely will find a higher % of your emails land in spam, or your content ranks a bit worse, etc.
There's a somewhat random continuum. Haunting is a funny word that does sort of include some variability.
Haunted implies a supernatural condition that just isn't helpful in system administration.
If something isn't working with a service there is always a method to troubleshoot and isolate the issue. Contact the appropriate people when needed. This is how NeoTokyo restored his "listed" domain.
It's not getting SEO blessings, true, but it's not disappeared.
So, haunted then?