Best Practices for Using Webhooks
stripe.com
stripe.com
As a person who uses webhooks from a variety of api’s there are upsides and downsides to them.
I suppose the same could be said for queues - but it strikes me there is a certain amount of “flimsiness” (if I can use that word) in just having a web hook fling packets at your endpoint until it finally dies.
There has to be a better way?
1. Keep a live connection to the queue, which is not great operationally (a lot of open connections just idling most of the time).
2. Poll the queue - at this point you're just polling.
3. Send an HTTP ping when something is added to the queue - essentially webhooks.
That's why a queue is not a silver bullet.
The key is these things need to have IDs and be idempotent always. Do that, and getting out of a lot of big screw ups needs no further effort/thinking after a reboot or rollback.
It adds a tremendous amount of complexity to all of your services, where now all of your services will need to maintain the ability to answer the question 'what was your state at this point in time.'
It makes it extremely difficult to support new webhooks, and honestly in my experience nobody actually ever queries for the historical data anyway. My opinion is that webhooks just aren't the correct approach to take for anything absolutely critical. They're a great addition for stuff outside the critical path where delivery is not essential, and as long as you stick to that paradigm you can dodge these issues entirely.
Re 'thin webhooks' which require hydration:
- subject to either complexity to support historical state (e.g. what was the state at the point in time when the webhook was issued) or race conditions (two things happened, but your API only gets current state)
- honestly none of your consumers want this - they're gonna be like 'why do i have to also query an API why can't you just hydrate the payload in the webhook body'. Guaranteed.
- still probably the correct solution if the data is super private and you want to be extremely cautious about issuing webhooks for it
source: worked exclusively on webhooks and similar eventing stuffs at place you've heard of for a few years
IMO webhooks are best used when your SLA for delivery is "best effort." Anytime you need to guarantee delivery (including guaranteeing the delivered data was correctly processed by the receiver), or guarantee anything about ordering or replayability, they just aren't that great.
What is your recommendation for situations that are absolutely critical? Is the recommendation to just have an /events endpoint?
I'm honestly curious because I have also implemented several wehbook-based systems over the past few years, and definitely seen the downsides, and so I'm trying to figure out the best way to manage asynchronous processing in distributed systems.
But it makes my life so much easier as a user of your service. I admit that it's not worth it for every type of service, but for financial ones it's 100% worth it. I'm trying to keep my ledger balances in sync with yours. I hate having to make a manual adjustment because something got missed.
Having my system heal itself even in the case when we really fucked up, pushed that bad update where we crashed and didn't persist what you said, or cloudflare fucked us and blocked you, or something... there's always something... I really really appreciate that.
I mean, if you don't because it's too complex for you.. okay, I'll have have to add the complexity on my side to compensate. I'm really thankful when I don't have to, and when I'm the event/truth source for others I try my best to do that courtesy for the event consumers.
Then you can add webhooks. Then you make the polling frequency slow as its job is really just to heal missed events while the webhooks provide real time notification.
As the service user you can use one or the other or both as you see fit. It's easy to get excellent performance and reliability.
Webhooks only is easier for the event provider but makes it hard on the consumer to recover from errors or outages. And pretty much everyone who offers webhooks fails to fire them or retry them sometimes.
I personally prefer the Event Hub (replayability, no HTTP request resiliency required, etc)
The problem with Event Hub is that it requires an SDK and if you've never used it it can be a pain. Also if you don't have an Azure account it's a problem (we typically provide our trusted partners an Event Hub on our subscription).
I also agree with some of the Webhook feedback here (better to also have some endpoint to poll for data/event verification and whatnot). At millions of homes, with a Thermostat being a REALLY chatty device, that's a lot of HTTP requests.
So, long story short, I prefer an event queue model over webhooks.
Shameless plug: One of the use cases of our product at https://www.inngest.com is reliably handling webhooks.
We accept a JSON payload, can optionally transform it using ES6 on ingestion, then do a few things:
- Fully type (and typecheck) the payload
- Store the payload in an event store
- Run N functions triggered by the payload, backed by a full idempotent queue (eg. your function can forward the event to your own API, and retry on failure)
This lets you do a lot: you get reliability because we'll continue to retry your API endpoint if it fails. You also get local testing, replay, and historical retries for free because we store the payload. For example, you can `inngest run --replay` from the CLI to replay historic events from your webhooks.
It's complex to build, but good event-driven architectures help with this.
Disclaimer: I built inngest :)
Edit to add: If you are interested in webhooks I am dusting off https://www.reddit.com/r/webhooks/ and would love to continue the conversation there as well!