How to safely invoke Webhooks | Hacker News Reader

How to safely invoke Webhooks

blog.fanout.io

60 pointsjkarneges17 comments

For Stripe's webhooks, we currently use the mechanism described in this post (resolve the IP address, munge the URL we're connecting to, and manually add a Host header), but we've run into a lot of problems.

If your client library supports SNI, you're likely going to send the wrong SNI hostname (we had to disable Ruby's SNI feature because sending an IP address in the SNI field broke a handful of sites), and cert verification generally becomes quite tricky.

We're in the process now of switching to a simple go-based HTTP proxy, which tries to pass through connections as faithfully as possible, but rejects connections to internal IP addresses. Here's the current implementation (though I'm still playing with it): https://gist.github.com/ebroder/ae9299e0078094211bde

This turns out to be way simpler - all HTTP client libraries have good support for proxies, and it doesn't interfere with anything about the SSL handshake (since HTTPS connections go through a CONNECT-based proxy).

We also looked at using Squid or Apache, but concluded that they were much heavier weight than we needed and it was difficult to make them behave as transparently as we wanted (e.g. they always want to add headers and such).

Great idea. A simple HTTP proxy with filter would be useful in isolating the logic and keeping clients simple.

The effect would be similar to what we do with Zurl (also described in the article), but more generalized.

Very cool, I'd love to see this published as a standalone tool, ideally with configurable blacklist IP ranges.

This is a good start, but misses two potential problems:

- The 172.16.x.x - 172.31.x.x address space, which is another internal-only one. Chances are that you'll know what your internal addresses look like, but things can change after something is built.

- What about IPv6?

It's really hard to get something right when you only have a blacklist to work with. It's much better to whitelist a few things that are allowed, but then that would limit the utility of Webhooks.

Other internal ranges: Agreed. We should probably put a formal list somewhere (maybe on webhooks.org) that all developers can refer to. Of course, there may be times when you do want internal targets to work as well. I want to say the problem should be something that people in charge of operations need to be aware of, and webhook-sending apps need configuration options to specify ranges. Then it's up to the expertise of the operator to make the right decisions. This doesn't feel very satisfactory though.

IPv6: Good question. I don't know enough about how it handles address spaces.

Fortunately, formally listing all of the internal IP ranges is a solved problem:

http://www.ietf.org/rfc/rfc1918.txt http://www.ietf.org/rfc/rfc4193.txt

I have a feeling that IPv6 is going to be a lot more complex than simply blocking FC00::/7. I seem to recall that one of the benefits of IPv6 was that NAT-ing would be a thing of the past and every system would have a public address.

What if I send you to http://lvh.me:10000/some/resource/?method=DELETE/ ?

edit: ok, it doesn't pass the socket.gethostbyname check

Indeed. You, know, though... I was almost afraid to click this from my personal machine.

Fortunately, links in a browser will always perform a GET, which hopefully is unlikely to do anything terrible on your own computer.

Links in a browser will always perform a GET, unless there's some JS attached to the click handler :) Fortunately I think HN is pretty safe.

Another option: route traffic through an external (authenticated) proxy.

Yeah, this is point 3 under the mitigation section. The only gotcha with this one is that you want to be sure there aren't any vulnerable local services on the proxy server itself. But this should be manageable.

It also may be possible to bind the connect port on the proxy and then use an iptables rule to prevent any connections from that source port from connecting to localhost.

The local port for outgoing TCP connections is chosen randomly in the normal case - and you can't have two TCP connections between the same pair of ports and hosts, so it would be hard to write a good proxy server that let you pick the local outgoing port.

You could use ipt_owner with iptables to block by source user matching the proxy server, however. Or just configure your proxy to block by outgoing IP if it supports this.

I think the poster meant IP not port.

Most proxy servers can be configured bind to a specific IP. So if your proxy server has an external IP and an internal IP, configure the proxy server to bind to the external IP. And then use IPTables to block any connections from the external IP to the internal network.

We run all our webhook invocations through our own product (a proxy that allows you to inspect any API call) which gives us this protection and gives us full logging to boot.

Interesting. Never implemented a service which uses Webhooks, but this is something I had never thought about. Useful to keep in mind.

I'm not a Pythonista, but AIUI the example code will still follow redirects, so internal pokes can still be made.

Thanks for the tip. I've switched the code to use Python's httplib rather than urllib2, which is more low level and doesn't follow redirects.

Reply on news.ycombinator.com