Hacker News Reader

Comment by dolftax | Hacker News Reader

We've been using API Tracker in production for few weeks now. The primary use case for us is to reliably handle webhooks from GitHub which our product relies heavily on (app installation, commit and pull request events).

Unfortunately, GitHub doesn't retry any failed webhooks and when our service goes down for a few seconds, thousands of webhooks fail and pile up. GitHub doesn't provide an API to query the failed webhooks and retry as well. We had to go through the painstaking task of visiting GitHub's app dashboard and click retry on each webhook, one by one.

With API tracker in place, we've updated our GitHub app's webhook delivery URL to send the webhooks to API tracker and they forward it to our services. In worst case when our service goes down for a while, API tracker gracefully retries all the failed webhooks.

Ref: https://github.community/t5/GitHub-API-Development-and/Handl...

View on HN

Interesting use-case for it. Without prior knowledge of a solution like this I would have suggested you send the webhooks to a queue backed notification system (e.g. SNS backed by SQS) and subscribe to the event topic, but sounds must easier to configure and manage the way you instrumented it. Might be a good use-case for me to try out!

This is something you can easily configure with our automatic retry function. We have an option to return a pre-configured response to the caller, and put the request in a queue to be retried until successful. This allows you to have a sustained outage while making sure all calls are eventually delivered.

> This allows you to have a sustained outage while making sure...

Re-driving queue backlogs at services recovering from sustained outages ends in tears almost always. Tread carefully. :)

Typically people use two pools for circuit breaking, with the limit set lower on retries: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...

Yeah, this is what I've seen most services who rely on webhooks from another service to do. Add in some monitoring of how many events are not yet processed (set a alarm when there is X amount of events in it) and you're done!

We're currently building a GitHub integration which receives webhooks and kicks off a bunch of processing actions based on the event type. Your suggestion sounds like a great way to add some observability to the service -- thanks!

> In worst case when our service goes down for a while

The worst case is still the same, no? API tracker goes down, GitHub has no redelivery, same deal. More a matter of whose uptime you trust more in this regard.

(That's not to say it's not valuable for this use case)

Sure. The least we expect from any service sending webhooks is built-in retry strategy. GitHub doesn't. We were thinking of building this ourselves internally but if someone takes care of this for you reliably, why not.

For API tracker, even if their services go down for a short while, it isn't good for business. Though it's been only few weeks using API tracker, we had zero failed webhook deliveries. They say they've designed their systems with this as a primary goal, of course. What if AWS or GCP goes down. It's a matter of trust and SLAs.

> What if AWS or GCP goes down. It's a matter of trust and SLAs.

AWS does have a 100% uptime SLA on some of its services: Route53, for example [0]. Not saying that ApiTracker could not be a 100% uptime service (in fact, it looks like that's their explicit goal), just pointing out that AWS / GCP do have services that never "go down" barring global catastrophes.

[0] https://aws.amazon.com/blogs/architecture/a-case-study-in-gl... -- Route 53’s foremost goal is to always meet our promise of a 100% SLA for DNS queries – that all of our customers’ DNS names should resolve all the time.

Thanks.

At $349 for 1M calls, doesn't it get expensive? I'd reckon, web-hooking it to Step Functions + AWS Lambda or SNS + SQS would have been a much cost effective solution at the cost of additional resources devoted to development and maintanence, of course. So, if you're comfortable sharing, what did the TCO economics look like for you when you decided to use ApiTracker instead?

Don't integrators like IFTTT already support GitHub integration? How is API Tracker different from IFTTT?

Reply on news.ycombinator.com