Webhooks Are Harder Than They Seem
svix.com
svix.com
The recommendation for "exponential backoff spanning several days" is particularly telling. What happens when your service faces extended downtime beyond their arbitrary retry window? Your "real-time" system suddenly develops permanent blind spots. Meanwhile, a simple polling implementation would seamlessly catch up by simply processing the backlog.
For mission-critical integrations like payment processing, I've found that polling Stripe's events API at regular intervals is not just more reliable - it's significantly easier to reason about and debug. When issues arise, you can simply reset your local state and replay events, rather than attempting to recreate the exact conditions that triggered a webhook failure.
The purported benefit? Saving a few seconds of latency in a business process where users are already accustomed to asynchronous workflows. It's architectural complexity theater that solves a non-problem while introducing very real failure modes.
The big components we ran into were:
1. Having retry logic if you don't get a 200 back. And some way of flagging the failed requests. We already had a pretty robust job queue/worker setup on our side, so it was a lot easier to add webhooks with that logic in place.
2. Keeping a webhook log. On our side we had a request history already that kept a log of our LLM requests, so it was pretty easy to reuse.
3. Auth / dev experience. Some way for a developer to put in a url + headers + auth method and click "test".
What means that, if you know your timing, miltiplicity, and reliability requirements and guarantees, you can solve most problems by taking the best design pieces from the state of the art (idempotency, write-only logic, retrials, optimist atomicity, peer election...) and applying whatever you can to simplify your code as most as possible.
If you know how, you can probably solve most problems just fine. But you must know how, and you must know the problem.
Not optimized for scale, likely only a few thousand per second capable.
With exponential/random backoff?
I wonder about an alternate timeline where someone had cornered the market on pub/sub as a service, a centralized pub/sub system that everyone goes through, kind of like how Google and Apple have a duopoly on push notifications.
Imagine this:
1. You are sending out emails through an email service provider (Mailchimp, Braze, etc) and you want to subscribe to an event stream of success/failures/errors/etc
2. You register with the email provider and they hand you back a token for SassyPubSub
3. Your backend service registers its token with SassyPubSub and lets it know exactly how you want to be notified - posts to a URL, over a persistent websocket connection, pushes to SQS/RabbitMQ, etc
4. The email provider pushes messages to SassyPubSub
5. You consume those messages however you want.
The flow should be dirt simple, just a few lines of code to setup. To be effective, it'd have to have industry wide standard adoption, but it would simplify so many things.