Show HN: Sequin – A Kafka Alternative to REST APIs
We support streaming to Kafka and NATS. If you don’t have a messaging system, you can use our serverless hosted one. To get your data at rest, you can stream to databases like Postgres.
I (Anthony) joined IFTTT in the early 2010s. I worked on a lot of integrations there. At the time, it was clear the landscape was a mess. Yet, more than a decade later, nothing’s changed.
A common misconception is that integrating is about making API requests. Or running background jobs that paginate endpoints. But that's just API extraction. After extraction is when the real integration starts. The integration is all the custom business logic that determines how your system should use that data or respond to it changing. [1]
Big companies have to integrate hundreds of systems. They use message buses. That’s because message buses are the best way to integrate. [2]
But what happens if the system is a 3rd-party API? How do you get data into your message bus? You need to extract.
Unfortunately, most APIs stop at REST. But REST sucks for extraction (have you tried detecting deletes?)
ETL tools like Fivetran and Airbyte help with extraction. But they're built for data engineers, not for developers. This is why ETL tools are not real-time, have poor support for messaging systems, are not programmable, and don’t support writes.
Integration platforms like Zapier and Tray.io can help you wire up external systems. But this means writing business logic in their WYSWIGs or DSLs. Developers want to maintain business logic in their code and tools (e.g. git).
The vision is to get developers to stop writing undifferentiated and tedious extraction code. We want every developer to be able to integrate with message buses – no matter the project. When a messaging system is your backbone, integrations throughout your system and across many external APIs all feel the same. Most of the differences between APIs are in frivolous details. The differences between ServiceNow and NetSuite's query parameters should not matter to you. When the API is just data, these differences disappear.
While APIs haven’t evolved much since the early 2000s, message systems have. Companies like Upstash, WarpStream, and Synadia offer serverless message systems. They’re cheap to get started with and easy to scale.
Along with partnering with these providers, we offer our own serverless hosted message bus [3]. This uses familiar message queue semantics.
Sequin uses a number of strategies for each API to extract data in real-time. You don't need to stub out APIs in your tests or setup ngrok to receive webhooks. Both test and dev just need to read messages off a queue or records in a database. And we offer a uniform interface for writes. Writes are committed to both the API and your message stream simultaneously, so you can “read your writes.”
Sequin's early, so it’s not perfect. We don't have any onboarding in our product. We want to add a lot more sources. We need better graphs and insights for observability and monitoring. But we pride ourselves on speed and consistency. We know that missing even a single event erodes trust in our event streams, so we work hard to eliminate that possibility.
We'd love to hear your thoughts and questions!
[1] https://martinfowler.com/articles/cant-buy-integration.html
[2] Using message buses to integrate goes all the way back to 2003's classic "Enterprise Integration Patterns" (Hohpe and Woolf)!
[3] https://sequin.io/docs/stream-apis/consume/overview - Uses NATS JetStream under the hood