Show HN: Airbyte, data integration platform with 300+ open-source connectors | Hacker News Reader

Show HN: Airbyte, data integration platform with 300+ open-source connectors

github.com

29 pointsArimbr13 comments

I'm a Platorm person who has been tasked with setting this up for some data shovelling for a data team at work. Would not recommend.

It's a pain to deploy/monitor and you'll have to abandon any hopes of reasonable/sane declarative configuration management. Everything eventually needs to be done through the UI, which is just buggy and painful.

Are you sticking with it or using something else?

Were you self hosting or using cloud?

We're considering this vs. Fivetran.

Hah. As if anyone listens to the concerns of Platform engineers. Product and data teams make their choices and we are left to pick up the pieces and glue them together into something passably functional.

We are self hosting. Don't think the cloud hosted version is being considered anymore now that I got something up and running.

Switched to Fivetran after experiences with Airbyte, operating Airbyte was extremely tedious, even primetime connectors had bugs that I personally had to fix, we keep Airbyte around for the "long tail" of SaaS tools that Fivetran doesn't have a connector for but managing an Airbyte deployment in a stable manner for the long term was not easy.

Know that there is also a CLI to manage configurations defined in YAML files. And a few options to deploy Airbyte in "one click". It's all in the README, sorry to hear you didn't find your way around our docs... There are a number of growing features and deployment options now.

We are using the CLI, but it leaves a lot to be desired. Honestly, if you want to get serious about configuration management you need to put out a Terraform provider ASAP.

The disaster recovery story in particular is very poor. To use this in production with confidence, we need to be able to spin up a brand new instance, and have a single command ("terraform apply" or equivalent), to apply the exact same configuration state (sources, destinations, connections) that was on a previously running instance.

Thanks for the feedback! Recently a community member created a Terraform provider: https://github.com/eabrouwer3/terraform-provider-airbyte

Thanks for the feedback and we (Airbyte) are actively working on IaC solutions. We are publishing a Public API in 2023 [0]. We are working on an official Terraform provider built on that API, as well as, language specific SDKs. We hope to have these tools ready in 2023 as well.

[0] - https://api.airbyte.com (There is a roadmap there).

Totally agree - in fact from the issues pages it seemed like there had been a conscious decision from Airbyte not to support a Terraform provider and instead build out Octavia, which as you say leaves a lot to be desired. Deploying Airbyte is still far too hard, I'd be happy to pay for an 'on prem' version like Gitlab where it's a bit more managed. I also think the failure modes for Airbyte on prem are too hard to debug in comparison to other data 'MDS' tools.

Thanks - just exploring Airbyte for a client so will be running through the motions myself. For another client, we just built our own data architecture to fix some of these issues and more importantly cut costs significantly.

Were you able to find something else self-hostable? Ideally, we really want to stay away from cloud.

I setup airbyte cloud when I was tight on resources and needed integrations done fast, had no bandwidth to spend a day or two working on the ETL flows.

Overall it's been good once it was configured. Ran into way too many issues for a paid platform though that gave me concern few people are actually using it in production.

I did spin up a local instance with docker compose and you get some integrations not available in the cloud hosted version.

I'm hesitant to give it a recommendation - their support is responsive and tries their best - eventually solving issues - but if something breaks and you lose visibility into marketing spend or something you NEED... it's not been 100% stable for me.

E: to be fair to them - APIs change and break sometimes - I get it, I manage like 40 APIs in our airflow - but taking 3-5 days to fix something you need for production is risky.

I don't think I've seen a YouTube badge on a README before. I like it!

Nice

Reply on news.ycombinator.com