Show HN: ConTube – A Scalable Data Connect Framework for Pulsar/Kafka Ecosystems

Hi everyone,

I'm excited to share a project I've been working on, called ConTube. It's a scalable data connector framework that aims to simplify and streamline the process of data migration between diverse systems. The key idea behind ConTube is to provide a universal set of data connection protocols, which can work with both Pulsar Connectors and Kafka Connectors, leveraging the rich data connection ecosystem of both systems.

For a quick overview, please check this example: https://github.com/RobertIndie/contube#example

One of the standout features of ConTube is its lightweight runtime. Unlike the Kafka and Pulsar Connect Frameworks, ConTube allows for running multiple tubes within a single JVM process or a single Docker container. This eliminates the need for running separate MQ clusters, which can be a game-changer in terms of resource utilization and operational complexity.

Moreover, ConTube provides a universal data format, capable of representing data from virtually any system. This feature is still under development, but the goal is to further enhance the framework's versatility and adaptability.

I want to stress that ConTube is still in the PoC stage. I'm hoping to get more invaluable feedbacks and support from the community. I would greatly appreciate any feedback, suggestions, or insights you might have. If you're interested in contributing to the development of ConTube, I would be more than happy to welcome you to the project.

Please bear in mind that while I'm proud of what ConTube has accomplished so far, I'm fully aware that there's still a long way to go. I'm committed to improving and expanding the framework, and I'm excited to see where our collective efforts can take it.

Here's the link to the [ConTube repository](https://github.com/RobertIndie/contube) for you to explore. I'm looking forward to hearing your thoughts and ideas.

Thank you for your time and consideration.