Using Facebook Notes to DDoS any website | Hacker News Reader

Using Facebook Notes to DDoS any website

chr13.com

308 pointskapkapkap76 comments

> In the end, the conclusion is that there’s no real way to us fix this that would stop “attacks” against small consumer grade sites without also significantly degrading the overall functionality.

Nonsense. Every web crawler should have some form of rate limiting. That's just good etiquette. I can control the number of requests that the Google search indexer sends to my site via webmaster tools. I don't see a good reason why Facebook can't be a good net citizen and do the same.

Especially, Facebook is not supposed to intelligently crawl websites but to just proxy images. That's an easy one.

Yes, it should intelligently proxy images.

Ah, but intelligently for whom?

Benefits to Facebook might outweigh intangibles like "being nice" to the people and places they are indirectly making money off of.

I think in this intance we've passsed the line between "not being nice" and "being a dick", or in this case being an accessory to a dickish move.

If the issue is used to DoS a site Zuckerberg cares about, not that I would encourage such action as it would in itself be a dick move, I'm sure some form of rate limiting will be implemented in short order...

I'm kind of wondering why they need so many different servers to fetch the file from the remote host.

It would be smart to at implement rate-limiting, and also delegate to a specific server close to the host to fetch the image, and then sync the image/file across their own networks to whatever server needs it.

It is just a huge waste of bandwidth that 100+ servers need to fetch the file, instead of Facebook itself absorbing the cost.

Well I would assume that when you add a link to a note the link gets thrown into a queue and a cluster of servers pops items off that queue to fetch and store the result. Also remember FB sees each link as a different link so it can't fetch it once and share it.

I remember reading that FB saw each link as a different link, but I somehow managed to forget that while writing the comment. Oops.

This is actually the third time this has been submitted [1] (including once by the actual author of this post). There's an extra `/?` appended to the url so HN thinks it's a different link. Not sure why - just adding the slash and question mark doesn't change the target.

[1] https://hn.algolia.com/#!/story/past_month/0/ddos%20facebook

Next post: using hacker news to DDoS any website.

If HN is still running on one server, I don't think that's going to be possible.

There's several hundred thousand computers in the hands of us readers.

Actually it has a name, "The HN Effect" (inspired by "The Digg Effect", I guess) and it's a real thing as many web sites go down when they are featured in the front page.

Or the Reddit alternative, "The Reddit Hug," which sounds quite heartwarming.

I've always seen it written as "the reddit hug of death". Not quite as heartwarming.

fyi, it used to be the slashdot effect before anything else.

Actually it has a name, "Slashdot effect".

http://en.wikipedia.org/wiki/Slashdot_effect

Theres so many local variations. In the german speaking part of the internet, there is the verb "geheist", after a popular tech news site.

Are you saying this is a bad thing? Personally I think it's a good thing that the duplicate detector is easy to get around. Allows for submissions to get multiple chances.

Yeah, the HN admins (including dang, very recently) have said that the detector is meant to be porous.

It isn't meant to stop you if you are intent on reposting a dup, it is just there for informational purposes and to make you stop and think about it before you really do it.

I don't get it. Why not just allow duplicate submissions, but show a warning and link to the previous submissions before confirming the submission? Or if you intend the URL trick to be semi-secret and thus limited to "power users," then just allow duplicate submissions for users meeting some criterion, like account age or a karma threshold.

first time I'm seeing this post, but also now I found out there is another portal for HN. Thanks!

Thought Facebook had already fixed it. I tried with an image on my domain and it only went till about 660KB/s.

Then I found a bigger file.

It's now maxes out my upload speed (50mbps on fiber). The notes have been deleted but Facebook continues to do http requests for the file. That, or Apache continues to write requests to the access log after they finished and Facebook does not close active connections when it knows that the answer will never be used.

Edit: Found a video (big file) hosted by Facebook. Guess who's under attack now :D

Edit2: Seems they're loading at about 1.6-1.9gbps speed, calculating from how quickly the images seem to 'load' (become blank 1x1px images) on my client and how big the actual file is.

I wonder, what is the legality of having them DDoS themselves, using something they've suggested will not impact on someone of their scale, and that they won't fix?

Are you breaching their ToC or AUP? (I'm sure I've "agreed" to it but I doubt I've ever read it in full.)

I have no clue and I personally do not really care :P

It's way easier to just fix the issue than to sue me, as long as I don't noticeably disturb their infrastructure (i.e. don't bother other users).

Why not just limit the number of request per site and per user? Ie. John Smith can only make X request (let say 1000) to www.google.com.

Doesn't seem too hard.

They state that 400-800 mbit/s == consumer grade sites? really?

When you're as totally awesome and uncaring about your customers as Facebook, you can toss out words like "consumer grade" to describe people affected by your fuck-ups.

Download fast and break (other people's) things.

Outbound traffic. That's only serving up a few large images a second. It's entirely IO bound on network output. Getting a 1Gbps connection is not a big deal.

I don't know what your setup is like but a connection != the requests my webserver can handle.

I know, it was two separate comments. A: It's "just" IO, performing probably the most heavily optimized IO pattern in the world (serving static files) and B: buying a 1Gbps uplink is trivial.

>"buying 1Gbps uplink is trivial"

1 Gbps dedicated uplink is not trivial everywhere in the world. You also need more quota. Facebook easily crawled more than 1 TB of data during an hour. The more the better may not be best solution here. The higher the uplink, faster Facebook may crawl. So instead of transfering 1 TB in an hour for a 1 Gbps uplink, the transfer will be 10 TB if you have 10 Gbps. This will also depend how much bandwidth they allow their crawler, there must be an upper limit though.

Serving 1Gbps of traffic from Amazon S3 would cost $250,000 a year.

Maybe that's not a big deal to you, but it would be to me!

Could one use this to attack Facebook from within its network, bypassing its DDoS mitigation measures?

I'll bet that if it could then suddenly they might start caring more about it.

I'll bet that could end you up in jail.

Wait... This comment thread looks familiar. Did this happen already?

I'm surprised they aren't going to give him a bounty for this. I also assume they realize that most reporters will post their rejected findings soon after they get denied.

If enough people start using technique they will have no choice but to create a fix for this.

https://www.facebook.com/whitehat

      Exclusions
      The following bugs are not eligible for a bounty (and we do not recommend testing for these):
      ...
      Denial of Service Vulnerabilities
      ...

That said, they do appear to be quite hypocritical when it comes to enforcing this, as at least one user on the /r/netsec thread claimed that Facebook paid them for a mail bombing vulnerability.

I guess you have to use this to attack high-profile targets (or Facebook themselves) until they notice.

It's not clearly stated. I think what they mean to say is DoS on Facebook using brute force mechanism as testing this would bring down the Facebook system itself. Now manipulating Facebook's server to cause the attack, that'd be different I guess.

I agree with you, forcing Facebook to be your zombie of doom is very different to attempting to DoS facebook.

If you can easily cause any website to become an on demand low orbit ion cannon, they're going to cause problems.

> If you can easily cause any website to become an on demand low orbit ion cannon

You can do this with any website that lets users post links which may or may not be automatically loaded (by other users too). There are tons of these around; forums, blogs, etc. See also: Slashdot effect. That's also a true DDoS, coming from many IPs, as compared with a few of Facebook's servers.

They were probably aware of the issue, as this kind of bug has been already published for Google [1] (by the same author, it seems) a month ago. Also, Google did not give him a bounty reward for his discovery.

[1] https://news.ycombinator.com/item?id=7371176

I actually found this bug first in Facebook and later in Google. Facebook said they needed time to think about this and I was hoping they would fix it, so I could not disclose this back then.

Join the club. I've reported it before as well, years ago. I even discussed it on my FB wall at one point with some tests against my own site. I never re-tested, figuring they'd take care of the issue. Guess not.

It doesn't really hurt Facebook.

Tying up these servers impacts them, makes people start blocking them, breaking this service.

Well if enough people point the images to a tarpit it might...

Same as http://chr13.com/2014/03/10/using-google-to-ddos-any-website...

so if Facebook and Google got this bug ... or feature ... point one to another and boooom goes the internet!

But isn't Google constantly "ddos"ed by definition? (I mean simultaneously used by almost every computer on Earth).

Ideally, if the webserver should just return HTTP 420 (Enhance your calm), 429 (Too many requests), or 509 (Bandwidth Limit exceeded).

Side note: 420 is not a standard HTTP error.

While this is true, it's also not a huge problem, exactly:

  > HTTP status codes are extensible. HTTP applications are not required
  > to understand the meaning of all registered status codes, though such
  > understanding is obviously desirable. However, applications MUST
  > understand the class of any status code, as indicated by the first
  > digit, and treat any unrecognized response as being equivalent to the
  > x00 status code of that class, with the exception that an
  > unrecognized response MUST NOT be cached. For example, if an
  > unrecognized status code of 431 is received by the client, it can
  > safely assume that there was something wrong with its request and
  > treat the response as if it had received a 400 status code. In such
  > cases, user agents SHOULD present to the user the entity returned
  > with the response, since that entity is likely to include human-
  > readable information which will explain the unusual status.

http://tools.ietf.org/html/rfc2616#section-6.1.1

This text was basically unchanged in httpbis: http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-2...

Now that they've been informed and chosen to do nothing do they have some legal liability if it's used in an attack?

To the people saying that this won't really hurt facebook, I beg to differ, this is a relatively simple to deploy DDOS attack, if enough people start utilising it, I can see a fair amount of Facebooks resources being tied up in these attacks.

They will only fix it when it starts to hurt them it seems.

This looks like a great free load testing tool! :)

This can be fixed easily ( i have read it fast, but i suppose this could be a fix)

Add a boolean to a attachment in the db, queryDiff

Add a string (for md5 hash)

Calculate the hash on every file with a query parameter, if the file is requested the second time (with different query parameters), check if the file hash is the same. If the file hash is the same, change the bool queryDiff

Next time you fetch the file, queryDiff is false, so you shouldn't fetch the url and only get the original one (which was already downloaded)

There's no way to know if difference in query parameter is a different image or not, you can't simply re-use a prior URL which is equivalent but for query param in place of a new one.

    http://example.org/images?image_id=12121
    http://example.org/images?image_id=7272

   http://example.org/images?image_id=12121
    http://example.org/images?image_id=7272

contain the same images, chances are very high that other query parameters will be the same.

If they are different, chances are very high that other query parameters are also different.

Want to up your chances, then instead of only 2 requests, raise the bar to 10 different query parameter checks and add some additional db values (like CountFetch:int,sameUntillCurrentFetch:bool)

As soon as sameUntillCurrentFetch = false, then you request all images in the future. If CountFetch = 5 and sameUntillCurrentFetch = true, then queryDiff = false.

It's kinda weird my answer was downvoted, any better solutions to avoid the stated problem then?

Calculate a md5 hash file to identify the image id

http://stackoverflow.com/questions/4032209/is-md5-still-good...

This is one of those “worse cast scenarios” when it comes to web application design...

So giving a 404 for unknown get parameters should fix this for your own site? A way for Facebook to detect such a thing would be to hash the images and when two images have the same hash and only differ by some get parameter it could remember that it can ignore that parameter.

Not a reliable check. If the first 2 are the same, then this would cause a false positive as the third, fourth and fifth could all be completely different.

Interesting. I wonder if being behind CloudFlare would mitigate this?

I'm really impressed digital ocean droplets can sustain ~900mbps

You can do the same thing with Google Docs...

So now Facebook can DDoS anyone they like, and claim it's out of their control?

Just use etags!

He's using "outbound" traffic as a DoS metric which is sort of novel. I guess it looked better than "1000 HTTP requests"?

Actually the number of HTTP requests is 180,000+.

An an HTTP request requires what, 3 packets inbound? Just saying calling it a "DDoS" seems slightly generous.

We also have to remember that these requests are shared between 100+ Facebook server.

DNS amplification works on precisely the same principle.

DNS amplification works by sending huge amounts inbound to a target and overwhelming them.

Reply on news.ycombinator.com