Hacker News Reader

Comment by schtono | Hacker News Reader

I'm wondering how long it will take until we see the first DOS attacks, triggered by 2 or more webhooks in a feedback loop (e.g. A triggers B triggers C triggers A triggers B ...).

Nice :)

View on HN

Good thinking! I guess we'll have to devise some way to avoid these kinds of cycles, or limit them by setting some type of maximum number of bounces on a webhook message.

That would be pretty difficult -- the API message itself isn't getting passed around so there's no way to add a TTL.

If a blog post triggers a Tweet, and that Tweet triggers a blog post, which triggers another Tweet....

How would you do that? The point here is, the webhook call can't really know what it's POST'ing to; all it takes to set up a loop is another web app that will itself formulate an arbitrary POST.

Maybe delaying subsequent POSTs from the same server, unless it's whitelisted, and Bayesing out fishy requests. That should reduce the load.

Just keep track of a 'url stack' in the POST message (assuming the post body is structured with something like JSON or XML). If a service detects a cycle it can drop the message or give back a 50x error or something.

The problem here is that you can aim a "web hook" at any other HTTP endpoint, which may or may not honor any "web hook loop avoidance" protocol you come up with, and if that endpoint re-triggers the activity that generated the hook update, you have a loop.

HTTP is stateless, so the "url stack" idea is going to be tricky to implement.

Yeah I suppose keeping a trail of breadcrumbs only helps you prevent accidental loops. A malicious service in the chain could just clear the stack and create a loop, and no one would be any the wiser.

A malicious machine that wants to DoS a server doesn't need web hooks in order to make a bunch of requests to the server. It can just run ab, the Apache Benchmark program.

Presumably DoSing wouldn't be the objective here. If some of your services in the loop send SMS messages or modify your checkbook register, that would be much more annoying than a simple DoS.

Basic http caching and rate limiting should cover this.

Unless it's hard to make new accounts, the best that rate limiting can do is disable the whole webhook service.

Your web server can't issue a 305?

Your load balancer can't enforce usage limits?

If adding new users is easy, then you don't have enough information to do targeted rate limiting.

You are talking up the stack, users, sessions, etc. I'm talking about using controls at lower levels of the stack; basic ip-based bandwidth throttling, forcing all requests to pass through a proxy, and the like. If the business requires that users be able to point random postbots at their account you have to plan for misbehaving counterparties; and application level controls while important will not be sufficient to keep the system up in the face of a broken client that is spamming the server with oversized requests at a pace higher than it can accomodate.

Even if you do low level throttling that maintains the stability of the system, it doesn't necessarily ensure usability. Who would want to use webhooks if there was much of a chance of frequently getting flooded by spam? It doesn't matter if the trickle isn't enough to kill your server, if you keep getting buzzed on your mobile every thirty seconds, you won't want to use the service any more.

This problem has been relatively solved for email beforehand, every once in a while the spammers figure out something new or something goes awry and you have a few offers for male enhancement products in your inbox; do you stop using email because of it?

The problem is solved for email because MTAs and mailing list programs go through contortions, documented in a long series of RFCs and Internet-Drafts, to avoid loops.

Also you might be able to get some milage from amplification ;)

A triggers B B triggers C and D

C triggers A D triggers A

Be quite cool to watch it happening - something triggers A, then pretty soon everything blows up.

It would be even cooler if the web could handle it.

Kind of reminds me of the "if a butterfly flaps its wings on one side of the world..." thing.

Reply on news.ycombinator.com