Hacker News Reader

We don't host user content under the gitbook.com, or at least we've stopped doing it a few years ago.

User content is stored under *.gitbook.io, similar to GitHub.

Google blocked all domains that contained "gitbook" in our account, even ones that are used for some infrastructure and are not accessible by the public. We don't know the exact reason for this, maybe they've blocked gitbook.com because we still have some redirect for content that was hosted under it years ago.

And yes we are going to make changes to host our status page under another domain.

That’s really important to add to the story! I think that makes the rest of the story more troubling. Especially knowing that domains unrelated to the apparent user phishing code were affected.

You might think of adding it to the post-mortem doc so that others don’t assume what I did.

why on earth would GitBook be hosting user-generated content under their corporate gitbook.com domain?

You mean like news.ycombinator.com, nytimes.com, facebook.com or most blogs or forums ever created?

Also their entire business is serving user domains so if that goes down customers don't care about their marketing site, though the email vanishing would be annoying. What's to stop google doing this with any regular domain with user comments or content though?

I blame the registrar - having heard this horror story I'd never host with google domains.

Facebook uses fb.com as their corporate domain.

Does it really make it less of a problem if they had a separate domain for corporate and it was only the user content going down? Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem.

Building and running anti-abuse tech works in proportion to the money put in. Can a small player really compete with Google’s anti-abuse investments? No. Save for a couple of exceptions, this means Google could always find abusive user content in its clients’ domains that the clients themselves couldn’t catch. I am not suggesting small players should be able to freeride Google’s anti-abuse capabilities, but if leads to enforcement and downtime of this sort, there needs to be a better protocol for the client to be able to respond or it paints a very anti-competitive picture.

> Does it really make it less of a problem if they had a separate domain for corporate and it was only the user content going down? Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem.

Yes, it really does. Not losing your ability to receive email will help resolve the problem.

> Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem.

If that is the case, then choosing a registrar that provides quality, timely and responsive support when dealing abusive content complaints should be on the top of your priorities.

> Building and running anti-abuse tech works in proportion to the money put in. Can a small player really compete with Google’s anti-abuse investments?

In this case the small company did find the content first through their own anti-abuse systems.

I'm not sure how this become anti-competitive. Google may have a monopoly on many things, but DNS registry is definitely not one of them.

> Not losing your ability to receive email will help resolve the problem.

Sure, I'm trying to prevent this stealing focus from the fact that this shouldn't have been a problem to begin with. The argument for separation is not made for defending against a SLA limitation or some other technical reason. It is made to accommodate a faulty policy problem. It is equal to saying "best practice: all big-co domains users should have corporate and production domain separation in case big-co's policy layer messes up. oh well". I'd rather see big-co held accountable not to mess up.

> If that is the case, then choosing a registrar that provides quality, timely and responsive support when dealing abusive content complaints should be on the top of your priorities.

Hindsight is perfect, while those qualities are rarely perfectly symmetrical information. "They should have known this could have happened" sounds like victim blaming to me. They were entitled to reasonable policies and reasonable application of those policies.

> I'm not sure how this become anti-competitive

Monopoly is not the only machinery of anti-competition. Creating a landscape demanding anti-abuse parity is essentially creating a barrier of entry. They are going to cloudflare next but there is no telling a similar scenario won't happen. It is important to call out the potential of big-co's putting capital-intensive demands on little players to participate in the web.

Where do you draw the line? I mean github.com is the corporate email domain of the GitHub company, it also hosts repositories (user generated content)

But not for any binary content as uploaded by the user, that's served from raw.githubusercontent.com.

I'm aware, but I could just as easily see a lesser registrar getting confused by an abuse report and suspending github.com because it embeds content from githubusercontent.com.

So you also use two different registrars (the cheap option)?

Or the more expensive option where you use a registrar that gives you get a direct phone number to an account manager.

Not all of it. Any bug attachments which don't have previews are served from https://github.com directly, for example.

Here, I uploaded that image from the other day that crashes some phones. I gzipped it so it wouldn't generate a preview, and attached it to a bug. When you click the "github.com" link, it downloads the file, and (at least with my web browser) uncompresses it and opens it with your default application. It's bit-for-bit the same as what I uploaded.

https://github.com/kengruven/strukt-bugs/issues/40

I don't know if this is exploitable. I haven't spent any time trying to break GitHub. This is just something I happened to notice once.

That's interesting. It looks like the link is active, but the request to github.com results in a 302 redirect to S3.

I don't know what that would mean in this type of scenario (phishing) either. I wonder what an html attachment would look like...

But it isn't... directly.

Github.com is their corporate site, hosts their application, and is how we all interact with repositories (via https or git://). But, the only data you get from that site has been processed through their application and sanitized.

The only way to get access to the raw user-generated data is through raw.githubusercontent.com or Github Pages which hosted on github.io. And the data from raw.githubusercontent.com has the MIME types set so that you don't get HTML rendered -- only the raw plaintext (I think).

So, in this specific example, if there was someone hosting a phishing site in a github account, it would have been active only through github.io, not the main github.com site. (You could have likely seen the source code from the main site, but it would not have actually generated an HTML form).

For historical views, here is the Github blog post that details the change (2013): https://github.blog/2013-04-05-new-github-pages-domain-githu...

Nevertheless some countries blocked github.com, not (only?) githubusercontent.com or github.io

https://en.wikipedia.org/wiki/Censorship_of_GitHub

Probably they think that GitHub didn't sanitize the content on github.com well enough.

That’s exactly what they are doing, too. Google effectively shut down github.com for the issues in github.io that didn’t exist anymore, in your example.

https://news.ycombinator.com/item?id=23425569

I can say from experience that moving content to a different domain without breaking things for users can be atrociously difficult. So if you make a mistake in how you split domains in the beginning, it can be very expensive to rectify later. Can you imagine going to all your customers and saying "you know all those links you have to example.com? You're going to have to change all of them to point to example.org now."

They _should_ split their domains, and given this recent issue, it would make sense for them to prioritize that. But, well, easier said than done.

I imagine it will be painful to split their domains. But, I was surprised to see that wasn't mentioned in their post-mortem. Even if they didn't plan on doing it, I think that it would have made sense to mention.

I imagine their main paying customers would start demanding this change though. They are probably using a custom domain, so whatever domain GitBook chooses to use should be irrelevant for them (in theory, but in practice, who knows...).

It most often has to do with browser sandboxing. See CORS, cookies, etc. It was a security flaw that began being addressed circa 2010.

Assumingly because GitBook didn't expect Google to be so reckless and tone-deaf.

because tech culture permits you to blame Google and their support team instead of knowing basic domain operation practices