Google Domains blocking all Gitbook URLS: post-mortem
blog.gitbook.com
blog.gitbook.com
Our production domains (gitbook.com and gitbook.io) have been blocked and locked by our registrar (Google Domains).
None of our infrastructure is impacted, all user content and databases are safe; our domains simply blocked by a heavy handed policy.
As mentioned on Twitter, we are all hands working with Google to fix this issues ASAP. We'll then share an in-depth post-mortem
https://twitter.com/GitBookStatus/status/1268554857411227648
$ dig gitbook.com a @8.8.8.8 +short
104.18.9.111
104.18.8.111We are working on making sure that everything is correctly working.
Workaround that we've setup to allow our users to still access the platform through different hostnames will continue working.
We'll share more details in the postmortem.
Are you sure that's the best strategy?
I am a fan of NameCheap personally.
I've been using them for years, and never had any technical issues...
Better than what Google Domains did to Gitbook.
Something similar happened to me – Namecheap dumped the wrong (private) information into WHOIS immediately after a redesign of their systems. It definitely was not user error.
Dealing with Namecheap's customer support to try to resolve this was possibly the worst customer support experience I've had in 20+ years in the tech industry. Lots of lies about getting back to me the next day, passing the buck, blaming everybody but themselves, extended periods of flat-out ignoring me, and eventually a complete inability to fix it.
I've been a happy user of Hover ever since, but I'm unable to recommend them – ironically because nothing has ever gone wrong with them. I used to recommend Namecheap until that nightmare happened, then I found out just how shockingly useless they are when it comes to customer support and privacy. Ever since, I only recommend services where something has gone wrong so that I know they are capable of resolving problems well. I regret ever recommending Namecheap and don't want to make the same mistake again.
Some more Namecheap horror stories here: https://news.ycombinator.com/item?id=18087862
It's cheap, at-cost, and they support a lot of the new TLDs like .io which is also a lot cheaper.
That to me is a downside since that means that that is not a core part of their business. Financially, it makes no difference to them if I use their service or not.
I would rather pay a little extra to a company that has domain registration as a core part of their business and actually makes a profit from me.
And domain names are cheap. Even if you pay twice as much as the cheapest service, it still will not make any difference in your bottom line.
I know the concern of putting all your eggs in one basket is real, but since CF's business is literally to take over your domain DNS and slap on some add-on services, adding domain registration in-house seems like a good fit.
You can avoid this issue by going with a registrar that focuses on bulk domain sales (eg. internet.bs in my case, but there are more, like eNom I think?), as they have a high-enough volume that they can easily stay afloat even when charging reasonable prices and without aggressive upsells.
It's mostly the consumer-focused "$1 for the first year" registrars like GoDaddy that you want to stay away from. Those are the really problematic ones.
> but since CF's business is literally to take over your domain DNS and slap on some add-on services, adding domain registration in-house seems like a good fit.
Sure, if you want to send all the traffic of all of your users through a man-in-the-middle US-based company with a very dubious past and a questionable business model revolving around basically centralizing the internet.
It's not a great recommendation to make. It also raises the question of why they seem intent on killing off the registrar market by offering "at cost" (which honestly isn't much lower than what aforementioned internet.bs charges anyway).
Net result: You get the domain at your preferred registrar, but you get 90% of the savings you would have got if you had it at the cheap register.
You might want to pick another example, .io is 23 years old [0]
Same with .ai fwiw.
https://priceonomics.com/the-rise-and-fall-of-ly/
https://www.theguardian.com/technology/2010/oct/08/bitly-lib...
Some random British company convinced IANA to let it run the .io domain for their own profit. Their operation of it has nothing to do with the interests of its exiled inhabitants (the Chagossians), the British territorial and military authorities, or the US military presence which constitutes the the territory's raison d'etre.
I think it likely that, one of these days, something is going to happen to the .IO ccTLD operators. Their rights to it are very dubious, and someone else (the British government, the government of Mauritius, the Chagossians) could end up wresting it from them.
(Disclaimer: I work there)
apple.com, twitter.com and ocado.com use CSC Corporate Domains [2]
I have no idea what such services charge, but they're all "call for pricing" and none of those companies would blink at spending $10k/year on their domains.
Not every well known brand uses such a service, though. bbc.com uses tucows, stackoverflow.com uses name.com and ycombinator.com uses gandi. facebook.com uses RegistrarSafe, a subsidiary of themselves, and almost every domain registrar is registered with themselves.
[1] https://markmonitor.com/ [2] https://www.cscglobal.com/global/web/csc//micro-domain-name-...
How recent is your experience with MarkMonitor?
The same message could also be worded more like "once you get past this, I'm sure you're already considering moving registrars. But please let us know if the support you're receiving from them is as bad as (my experience / reputation / etc.)".
Because 99% of people don't realise that their domain registrar holds the keys to their business's entire internet presence.
They can switch off your website/email at any time, with no real consequences apart from a little bit of bad PR if you have enough social media followers or post in the right forums where their staff hang out.
They can also do a shitty job of securing your domain, and let it get stolen/hijacked. The attacker then gets to set up their own MX records and collect all the password reset emails they triggered on every other important site, and pretty much own anything you doin't have 2FA set up on.
Anyone who doesn't think customer support from their business domain registrar is a thing worth paying for, most likely hasn't evaluated the risks properly.
...I don't host any DNS at Google, and if this is actually not spin, now I never would.
Not that I trusted Google before, but good lord, this is egregious.
let us know if you have any questions!
I know it moves the rug under the existing discussion, but it's better than having two separate threads.
Cloudflare: "Cloudflare and Registry Operator may deny, cancel, suspend, transfer, redirect or modify the Registrar Services or a Registration, or place any domain name(s) on lock, hold or similar status, as either deems necessary, in the unlimited and sole discretion of either Cloudflare ... for distributing malware, abusively operating botnets, phishing, piracy, trademark or copyright infringement, fraudulent or deceptive practices, counterfeiting or otherwise engaging in activity contrary to applicable law."
Google: "We may in our sole discretion, deny, suspend or cancel any registration or transaction, or place any domain name(s) on registry lock, hold or similar status if ... engaging in spam, phishing, or other deceptive practices."
EDIT: I see you're moving to Cloudflare, but I wish you the best of luck
Being forcibly stuck on CloudFlare's own nameservers only sounds very nefarious, and isn't a limitation I've ever heard of with any other registrar. For instance, it would break my tooling that uses my host's APIs to control DNS records through their nameserver.
I'd be very appreciative if eastdakota or jgrahamc could elaborate on what possible reasoning there is for this restriction as well.
Doesn't seem conducive to great customer support, but maybe I'm wrong cause I've never had to contact them.
The registrar issue is one problem here -- but I don't see this addressed in their post-mortem. I thought this was a well-known issue. You don't host user-generated content on the same domain that handles your corporate email. Because things like this can and do happen, so you want to make sure that your company isn't also down while you're busy fixing a problem with your customers.
I know the exposure risks are different, but isn't this is one of the reasons why Github moved hosting Github Pages content to github.io from their primary github.com site? Or why raw data is hosted from githubusercontent.com (in addition to mitigating cookie security issues).
The registrar was an issue that was largely out of their hands. But this was something that they could control. And I think it's something that is missing from their post-mortem. If they had split their domains, then while serving user-content was disrupted, accepting users to gitbook.com and their email (!?!?) would have still been working. Also, depending on if they had a 3rd domain for hosting their CDN, users that had custom domains would also have been protected.This goes along with the idea that you don't use email from your primary domain (cto@gitbook.com) to register your domain (gitbook.com). Or host your status page on the same infrastructure as your site.
If I were them, after migrating registrars, this would be the next engineering change I'd make.
(Maybe they do this, I don't know enough about GitBook to know. But based on the thread here, I don't think they do, or at least it isn't mentioned in their post-mortem that I saw.)
Or the more expensive option where you use a registrar that gives you get a direct phone number to an account manager.
Here, I uploaded that image from the other day that crashes some phones. I gzipped it so it wouldn't generate a preview, and attached it to a bug. When you click the "github.com" link, it downloads the file, and (at least with my web browser) uncompresses it and opens it with your default application. It's bit-for-bit the same as what I uploaded.
https://github.com/kengruven/strukt-bugs/issues/40
I don't know if this is exploitable. I haven't spent any time trying to break GitHub. This is just something I happened to notice once.
I don't know what that would mean in this type of scenario (phishing) either. I wonder what an html attachment would look like...
Github.com is their corporate site, hosts their application, and is how we all interact with repositories (via https or git://). But, the only data you get from that site has been processed through their application and sanitized.
The only way to get access to the raw user-generated data is through raw.githubusercontent.com or Github Pages which hosted on github.io. And the data from raw.githubusercontent.com has the MIME types set so that you don't get HTML rendered -- only the raw plaintext (I think).
So, in this specific example, if there was someone hosting a phishing site in a github account, it would have been active only through github.io, not the main github.com site. (You could have likely seen the source code from the main site, but it would not have actually generated an HTML form).
For historical views, here is the Github blog post that details the change (2013): https://github.blog/2013-04-05-new-github-pages-domain-githu...
https://en.wikipedia.org/wiki/Censorship_of_GitHub
Probably they think that GitHub didn't sanitize the content on github.com well enough.
Building and running anti-abuse tech works in proportion to the money put in. Can a small player really compete with Google’s anti-abuse investments? No. Save for a couple of exceptions, this means Google could always find abusive user content in its clients’ domains that the clients themselves couldn’t catch. I am not suggesting small players should be able to freeride Google’s anti-abuse capabilities, but if leads to enforcement and downtime of this sort, there needs to be a better protocol for the client to be able to respond or it paints a very anti-competitive picture.
Yes, it really does. Not losing your ability to receive email will help resolve the problem.
> Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem.
If that is the case, then choosing a registrar that provides quality, timely and responsive support when dealing abusive content complaints should be on the top of your priorities.
> Building and running anti-abuse tech works in proportion to the money put in. Can a small player really compete with Google’s anti-abuse investments?
In this case the small company did find the content first through their own anti-abuse systems.
I'm not sure how this become anti-competitive. Google may have a monopoly on many things, but DNS registry is definitely not one of them.
Sure, I'm trying to prevent this stealing focus from the fact that this shouldn't have been a problem to begin with. The argument for separation is not made for defending against a SLA limitation or some other technical reason. It is made to accommodate a faulty policy problem. It is equal to saying "best practice: all big-co domains users should have corporate and production domain separation in case big-co's policy layer messes up. oh well". I'd rather see big-co held accountable not to mess up.
> If that is the case, then choosing a registrar that provides quality, timely and responsive support when dealing abusive content complaints should be on the top of your priorities.
Hindsight is perfect, while those qualities are rarely perfectly symmetrical information. "They should have known this could have happened" sounds like victim blaming to me. They were entitled to reasonable policies and reasonable application of those policies.
> I'm not sure how this become anti-competitive
Monopoly is not the only machinery of anti-competition. Creating a landscape demanding anti-abuse parity is essentially creating a barrier of entry. They are going to cloudflare next but there is no telling a similar scenario won't happen. It is important to call out the potential of big-co's putting capital-intensive demands on little players to participate in the web.
They _should_ split their domains, and given this recent issue, it would make sense for them to prioritize that. But, well, easier said than done.
I imagine their main paying customers would start demanding this change though. They are probably using a custom domain, so whatever domain GitBook chooses to use should be irrelevant for them (in theory, but in practice, who knows...).
You mean like news.ycombinator.com, nytimes.com, facebook.com or most blogs or forums ever created?
Also their entire business is serving user domains so if that goes down customers don't care about their marketing site, though the email vanishing would be annoying. What's to stop google doing this with any regular domain with user comments or content though?
I blame the registrar - having heard this horror story I'd never host with google domains.
User content is stored under *.gitbook.io, similar to GitHub.
Google blocked all domains that contained "gitbook" in our account, even ones that are used for some infrastructure and are not accessible by the public. We don't know the exact reason for this, maybe they've blocked gitbook.com because we still have some redirect for content that was hosted under it years ago.
And yes we are going to make changes to host our status page under another domain.
You might think of adding it to the post-mortem doc so that others don’t assume what I did.
I don't expect google appreciates how devastating this, but that would change quite immediately if it happened to them.
I've always assumed that:
1. When creating your account at the registrar you should use an email address that is not at any domain you will be registering through or transferring to them,
2. The contact information you give for your WHOIS records (or if using a WHOIS privacy guard service, the contact information you give them to forward to) should use an email address at a different domain,
3. If you use a DNS provider other than your registrar the contact email address they have for you should not be a domain using them for DNS, and
4. Same for whatever hosts your email.
The general principle is that the contact method that a service provider will use to contact you if there is a problem with your account or service with them should not depend on the account being in good standing and the service working.
Many of those are long gone, but PhishTank hasn't cleaned them out, so they're still listed.
[1] http://www.sitetruth.com/reports/phishes.html [2] http://www.sitetruth.com/fcgi/ratingdetails.fcgi?details=tru...
I run the service at https://urlscan.io which tracks phishing and frequently run into these cases which render any kind of black/whitelisting impossible. Imagine Microsoft phishing hosted on Microsoft domains and infrastructure. Here's a fun search which will return lots of phishing on windows[.]net and googleapis[.]com: https://urlscan.io/search/#page.domain%3A(googleapis.com%20O...
I used to contact nonprofits and small businesses which showed up on that list. Inevitably,they'd had a break-in. With some nagging, I could cut the size of the list in half.
I find that a lot of phishing sites these days are hosted on legitimate sites that have been backdoored, but are continuing to operate as normal.
The phishing page is hidden away in the /.well-known/ directory or another similar one, so the website owner is non-the-wiser.
The big difference here is that all the other platforms struggling (and failing too often) to prevent themselves being used as phishing hosting - aren't then turning around and hypocritically taking other platform's entire internet presence offline for doing so.
Google here are acting in the roles of judge and executioner, while being the biggest offender of the same crime.
Coupled with the horror stories of non-existent support, the first thing I did was move my domains out this month.
It's got all my eggs in one basket - but solving that problem is a much bigger task that just picking a different domain registrar... (And I sometimes wonder if my blue-sky cloud-agnostic dreams would all come crumbling down if I even managed to implement then anyway - if whatever went wrong that pissed AWS off enough for them to shut me down got shared with and/or triggered the same reaction at Azure/Google/DO/whoever...)
Currently don't have any hosting requirements but if/when I do, I either use AWS or digitalocean.
Had major problems with a trial. Firstly it would just redirect back to the admin console if I hit gmail. Support couldn’t fix it.
Also you can’t give anyone access to YouTube (!) for 60 days after signing up or paying $30 in credit. Which is no good because when you pay them £30 fuck all happens. And again support were useless.
Happy O365 customer now. And I’ve used support which was excellent.
Honestly I've tried everything and can't find any support pages. In order to get support you need to login to admin console on G Suite, well how the heck am I supposed to do that if I can't login to begin with? This is the generic https://imgur.com/a/lgRby50 page I get. It's been almost a week and a half, more than 5 business days, and I haven't heard back from my request to restore my account.
The problem is that they have my data and my credit card for this G Suite account.
I assume Office 365, from Microsoft.
Since then every-time I have the opportunity I strongly oppose any business I work with to move to any Google cloud service and I was pretty convincing so far in large corporations, and I will continue to do so.
As other posters have pointed out, hope they have a plan to migrate their domains elsewhere.
Several years ago I ran into almost exactly the same issue with them after someone sent them a frivolous abuse report - they'd locked down my domain unannounced to the point that I couldn't even transfer it out, and it took a call from a journalist(!) inquiring about the suspension before they were willing to unlock it for transfer.
I've been using internet.bs for most of my domains since and haven't had any issues with them - they let me know when an abuse report comes in, and give me the time to handle it. There are probably other registrars that are fine too, but I don't have personal experience with them.
https://twitter.com/GitBookStatus/status/1268565887256330241
I used to use a hosting company that had in their terms, that profanity wasn't allowed... The company seemed to have decent hardware, user interfaces and US based support too but ran by Mormons based in the beautiful state of Utah. I don't have much of a opinion of Mormons but I think pushing religious views on your customers is bad business.
So after noticing that in their terms I was curious since I run a personal WordPress blog, if someone wrote a comment even if it wasn't approved just by being in the database if it broke their terms and was told yes... Now that I'm a bit older, I wonder if they really are scanning database table text fields for swearwords or that support rep misunderstood what I meant... But after that discussion, I switched my domain and hosting elsewhere, two separate companies actually instead of just one account with both domain and hosting. If some spam bot or troll writes the F word, they are going to take my entire site down even if no fault of my own? Yeah, no thanks.
They got acquired though by a lot larger company and doesn't seem to have those same terms now though, and heard they now outsource their support. There was someone I used to talk to who was a big fan of them though and referred me. So slightly over a decade later, they are probably not even remotely the same company anymore though since merged with a much larger company. Wouldn't surprise me if they ended up getting rid of a lot of support staff and just kept a handful of people to manage the datacenter, since marketing, accounting, support, etc would probably be centralized between all the hosting companies they own I'd imagine, which is probably a blow to the local areas when they merged unfortunately. I guess that's one of the sad things about getting bigger, they lose that small town friendly startup feel probably as seemed like a great company other than forcing religious views on people. Just did some more research about their company, sounds like they were also anti-lgbt too sadly. Makes me wonder how many other people working their share those views or just something management was pushing down since looks like the church changed their policy last year... I've been wanting to go somewhere with more opportunities, and actually Salt Lake City is one of the areas I've been considering since seems like a bit bigger city and on the list as a place for upcoming startups.
After 3 sudden blocks from them, they now give us a warning before they're planning to block us, but it took many calls before we got to that stage.
"I built my business on Google Cloud / Android / Google Apps / Youtube / etc. I was flagged by an automated system. There was no human to speak to (at least one competent or empowered to do anything), and my business is now gone."
Some of these go viral on social media, and Google then fixes them. Some don't.
Support channels built on only having an impact if things go viral aren't what I'll build my business on.
I wonder how the domain registry community at large feels about this? ICANN exists, domains are subject to a legal agreement with ICANN, and it has customer-protection concerns surely?
Their support staff said that due to ICANN policy, they must have accurate WHOIS info, so they have changed my WHOIS information but also added WhoisGuard to the domain so the details are not visible unless someone gets a Panama court order.
My trust in them increased substantially.
I don't expect many startups or big companies use Google Domains for DNS hosting. AWS Route 53, Google Cloud DNS, or dedicated DNS hosting would be used for that. But I expect it is fairly common for startups to use Google Domains for domain registration.
Gitbook appears to use CloudFlare to host DNS, and likely to use their CDN/proxy, which makes them the actual "host" of the content. If there is any phishing, they should be the ones responsible.
The post says even their email was down. I'm deciding to not use Google Domains anymore.
Some might consider that sites with controversial content who use Google as their registrar and not taken down are considered reviewed-and-OK by Google.
Others might start pressuring Google to expand the nature of content they find unacceptable.
And yet others might try pressuring them to expand their policing - who knows how many bad actors could be shut down with Gmail monitoring?
Too bad Google does a terrible job of it. No way am I signing up for a Google Domain in the future.
The point is that there is a large lobby out there for knocking different types of content offline, and they don't show much concern for whether their demands also seem reasonable.
Most registrars feel they should not be in the content policing or law enforcement business. Google volunteering to do it in one area that makes sense opens the door for pressure to do it in other areas that make less sense.
To be clear, I think the Gitbook team actually handled this very well and it wasn't their fault, but it does illustrate very clearly why we shouldn't be relying on cloud services when we really don't need to.
* Fewer security and compliance hurdles when its on our own infra.
* Perfect uptime because it’s our responsibility. (Not saying we’re better but outages are our fault and don’t make you look bad)
* Fewer hurdles when integrating LDAP and friends. There’s no need for the weird “connectors” that companies come up with.
* We’ll pay you more for the privilege of hosting it ourselves.
* Giving us access to the code in a shared source model buys you an extreme amount of lock-in since once we start tweaking our fork for our needs we’re not leaving.
* The support burden is higher, sure but if it’s your most expensive enterprise tier you’ll almost always be working with a professional IT team.
* We’re the easiest crowd to up sell and we’ll even pay you to develop features you can sell to other people.
I don’t want to make it seem like selling in enterprise isn’t a slog but it’s a good business to be in.
I remember logging into an Ad Exchange account and discovering Google was sending emails to it. No one at the company was aware of it.
All the popular dedicated domain registrars I have used so far have excellent human support. Godaddy, namecheap, namesilo to name a few. I don't know if big companies or corporate use something more to secure their domain names and DNS, do they?
I think that, despite what frequenting HN may make you think, most people using Google's services - even the more complex or paid ones - aren't aware of the problems with support.
I always assume the people complaining about nonexistent support from Google are trying to get support for something they aren't paying for. You pay for Domains, and the support reflects that. You probably can't get support for getting locked out of a consumer Gmail account or help uploading a YouTube video.
The support agent couldn’t do a single thing but tell me to wait for the possibly robotic appeals process.
If you are using the registrar's built-in DNS hosting, move away from this first, which can also be done with no downtime.
Besides, any user-content serving platform will have to deal with malicious users. It is not a perfect process, especially against bot traffic. Shutting down the whole domain was very heavy handed.
I realize some things take time, but the onus is on them to do a final verification before flipping the switch to shut down an entire business(!)
Google is "saving money" on support costs while simultaneously destroying their brand image (at least among the tech crowd). It will be near impossible for them to re-earn that goodwill. It's such ridiculously short sighted thinking.
I'm moving off of G-Suite this week. This is the final straw.
Proactive spam control is not a solved problem anywhere. Banning a website over it is absurd.
And any time support or PR is droning on about "muh policies" you know there are problems.
I've seen domains like google5[.]$tld which contained a fake Google Login form, up for close to an hour with zero detects in VirusTotal and no detection by Google Safe Browsing itself. If Google fails at detecting phishing against their own brand, what chances do smaller shops realistically have? Here's the example I mentioned: https://twitter.com/urlscanio/status/1178043405529763841
It seems like a problem as hard as detecting spam.