Comment by tasn | Hacker News Reader

Comment by tasn | Hacker News Reader

Hey everyone, my name is Tom, and I'm excited to be launching Diahook on HN!

Diahook makes it easy for developers to send webhooks. Developers make one API call and we take care of deliverability, retries, and offer a great developer experience for their users. Essentially, we make it possible for everyone to offer a Stripe-like webhooks experience.

At my previous company, our users were constantly asking us for webhooks, both for consuming in their own services and for integrations with no-code solutions like Zapier. However, we kept on deferring building them because we weren't willing to commit the engineering time, resources and ongoing maintenance required of a webhook delivery system.

There are a variety of challenges when it comes to sending webhooks. For example customer endpoints fail or hang much more often than you would think, so you need to implement retries, but also make sure that such failures don't slow down or block your send queue or the rest of your system. Additionally, because of how webhooks work, anyone can send fake webhooks to your customers, so you need to make sure to cryptographically sign the payload, and make it easy for your users to verify it. You also want to avoid overloading your users' endpoints, so you want to automatically rate-limit webhook sending, as well as disabling failing ones, and notifying your users when you do.

I love webhooks, and I think everyone should be offering them! Our goal with Diahook is to make it faster for developers to add webhooks to their service, take care of the above challenges (and more), relieve them of having to worry about maintenance and scaling, and offer their users a UI for inspecting, debugging and replaying of past webhooks out of the box.

I'd love to hear about your experience building (or using) webhooks systems. What's a must have? Any war stories to share? Got any questions? Suggestions? Please let me know!

Docs: https://www.diahook.com/docs/

API viewer (and OpenAPI specs): https://api.diahook.com/docs/

>Developers make one API call and we take care of deliverability, retries

Don't developers need to handle deliverability and retries to Diahook?

Agree, I feel like anything like this as a service can be helpful, but misses the mark on use case.

I don't think they do, our whole job as an API company is to have uptime (and redundancy for uptime). I think it's the same with every other API company, such as Sendgrid, Twilio and etc.

User endpoints on the other hand, fail all the time, and often require a few retries.

With that being said, we plan on having client side (in our libraries) redundancy (try another endpoint if one fails), and ways for you to gracefully handle errors locally for users who need this.

Edit: I realise this comment was a bit sloppy, added some context in another comment https://news.ycombinator.com/item?id=26400970

It's not just your infra though. Say I'm a client sending you webhooks. My internet connection (if not in cloud infra) could be down, or my cloud provider might not have a route to your delivery endpoint due to a routing or similar outage. If those requests block, the client's app is going to hang or back up. If they don't block, they have to go somewhere (perhaps journaled with a task that can be executed on demand or on a recurring schedule).

My suggestion would be not only a library, but a local queue that can be managed with the library. Also, having some experience with working at a no code provider previously, ensure that it's trivial to query your service for what webhooks you can confirm were received by your infra and then delivered, perhaps by UUID.

Last, make sure it's dreadfully difficult to make changes to your systems where data loss could occur (messages dropped while returning 200 to counterparty systems, for example).

It's a good point, and actually some of our users already have this local queue implementation anyway because they want to send webhooks asynchronously. So it's definitely something to keep in mind. We already make it easy to query the service for what webhooks were delivered, but we also let you use the API with idempotency keys to ensure messages are not accepted twice.

I agree about all the issues you outlined though. One thing we plan on doing in terms of internet being down and cloud provider not having routes: have a lot of endpoints spread geographically in every region, and maybe in the same/nearby data centers.

Thanks again for the feedback!

For what it's worth, Stripe's Idempotent Requests are a nice way to handle this: https://stripe.com/docs/api/idempotent_requests

We just send the same requests over and over until they're successful. It's relatively simple to use.

Yeah, we also support idempotency (the event ID is used for that). I'll add it to that comment, I forgot to mention it. :P

Edit: ahh, I already mentioned it, it's buried in my comment.

> I don't think they do, our whole job as an API company is to have uptime (and redundancy for uptime). I think it's the same with every other API company, such as Sendgrid, Twilio and etc.

That doesn't sound realistic to me. Sendgrid, Twilio, and all other APIs have downtimes and other problems, just check their status page: https://status.sendgrid.com/history.

I always had to implement some retry logic in API clients at some point or another. That doesn't mean your service doesn't bring value but it's not enough to just expect it to work because it is focused on uptime. When implementing a service that requires external dependencies I would always expect them to fail and try to design with that fact in mind.

I agree, and I think my original comment was a bit sloppy.

We already have users that have their own queue system for sending us messages because that's how they deal with similar APIs too.

So I definitely see the problem, what I meant in my comment is that it's similar to other API companies, and it's not unique to us!

It's not always your service crashing you need to worry about; reliability also includes the network (and all of the routers) between you and your customers.

Yeah, I replied to a sibling comment. You are right, I was a bit sloppy with my message. I was just trying to make the point that it was similar to other APIs, and that we take uptime seriously.

Congratulations on launching this much-needed product! I've rolled my own webhooks several times, so I personally know all the pain points you mentioned.

I work in healthcare tech, and in the last couple years I've worked with legacy medical systems that use FTP to implement "pseudo-webhooks". For each integration with a third-party technology provider, there is a dedicated FTP server that reads/writes files to an FTP server owned by the third party. When this service is interrupted there is no retry logic - someone has to spend time manually checking if files made it across.

If someone could make this legacy process easier on developers and HIPAA compliant they would be rewarded handsomely, as these organizations (hospitals, medical device makers, health-tech companies, etc) are comfortable paying enterprise-level prices for any technology they use.

We are going to start working towards HIPAA soon, and we also going to offer a way to end-to-end encrypt the payload in the very near future. The encryption doesn't help with HIPAA (does it?), but it definitely helps with peace of mind and reducing the attack surface.

Very interesting. What’s the best way to learn more about this problem? Are you working for a healthcare tech company integrating with these legacy systems via FTP, or are you working for a provider or insurance company’s IT organization trying to integrate with said legacy systems for in-house software?

Is this what it would look like? https://docs.mulesoft.com/ftp-connector/1.5/

If there’s any links or vendors you could share so I could learn about this I’d greatly appreciate it.

Happy to elaborate more for you. I work on the health tech side, so we’re integrating with legacy systems. I’d say there isn’t any standard for the actual mechanics of how the data is interchanged, but usually organizations adopt the HL7 standard for payloads. Most of our integrations look like “we’ll send you this HL7 payload, then we’ll periodically check this folder for any HL7 payloads to stick into the patient record”. If you want an analogy, think of any vehicle that human lives are dependent on, held together by duct tape. :)

I will say there are EMRs like Athena Health which actually build for the developer experience. Integrating with Athena is relatively easy, and you can scale up the integration across customers easily. Epic (the market leader) is much harder and more expensive to do this for, since every distribution usually has some customization.

This is great =). I have owned a large webhook delivery system myself, and considered starting a SaaS around it. I'm also a big fan of webhooks and I think there is definitely a market for this out there. Kudos to you for taking the plunge and launching!

Here's the only post I wrote for it that focuses on security, which is pretty critical for a webhook system https://www.easywebhooks.com/how-to-secure-a-webhooks-api. You might want to consider adding protection against Replay Attacks and support for Challenge-Response Checks if you do not already!

We already protect against reply attacks (we have a signed timestamp as part of the webhook), and I'm not sure what you mean by challenge-response checks. Do you mean things to protect against SSRF? If so, yeah, we also sign the webhooks in order to make sure they come from the right source.

Thanks for the link, will read it tomorrow!

A challenge response check will help ensure that the webhook consumer is actually using the webhook signature, and improve the likelihood that you are sending data to the right target. I saw a number of times that systems weren't verifying the data we were sending them came from us even though we had gone to all of this effort to help them :facepalm:.

Basically you periodically send a GET request to the target API with a token, and have them respond with the token encrypted with the same secret they'd use to decrpyt your webhook signature.

You could also consider sending dummy ping messages that may or may not have a valid signature (of course make sure this behavior is documented) that you would expect the target API to return a 4xx error if the signature is incorrect.

These extra steps are definitely not table stakes for a webhooks system, but could be enough to make sure the webhook event providers are being the best possible stewards of their user's data that they can =D. A lot of this complexity can also be wrapped by a client library you provide, which is a big win for everyone on its own.

Twitter's webhook API has an example of CRC btw! https://developer.twitter.com/en/docs/twitter-api/enterprise....

Ah I get what you mean now! Interesting way to ensure they are actually doing the right thing! I think the ping with the bad signature to check they are actually verifying the webhook is much lighter weight, and almost solves the same problem. The one with the signed response is also interesting, but a bit harder to get people to actually implement.

This is a neat idea! Harder than it sounds, but y'all seem to have most of the aspects of the problem in mind already.

My employer (Segment) has something similar to Diahook (as I understand it) at its core: https://segment.com/blog/introducing-centrifuge/

I just skimmed through the post (it's late here, haha). It looks like Centrifuge is cool and is a part of what a webhook systems needs, but not all of it. Check out the landing page for some of the other things we provide.

Thanks for sharing this though, a cool read for tomorrow!

Reply on news.ycombinator.com