Hacker News Reader

Comment by bsnnkv | Hacker News Reader

I'm a Platorm person who has been tasked with setting this up for some data shovelling for a data team at work. Would not recommend.

It's a pain to deploy/monitor and you'll have to abandon any hopes of reasonable/sane declarative configuration management. Everything eventually needs to be done through the UI, which is just buggy and painful.

View on HN

Thanks - just exploring Airbyte for a client so will be running through the motions myself. For another client, we just built our own data architecture to fix some of these issues and more importantly cut costs significantly.

Were you able to find something else self-hostable? Ideally, we really want to stay away from cloud.

Know that there is also a CLI to manage configurations defined in YAML files. And a few options to deploy Airbyte in "one click". It's all in the README, sorry to hear you didn't find your way around our docs... There are a number of growing features and deployment options now.

We are using the CLI, but it leaves a lot to be desired. Honestly, if you want to get serious about configuration management you need to put out a Terraform provider ASAP.

The disaster recovery story in particular is very poor. To use this in production with confidence, we need to be able to spin up a brand new instance, and have a single command ("terraform apply" or equivalent), to apply the exact same configuration state (sources, destinations, connections) that was on a previously running instance.

Thanks for the feedback! Recently a community member created a Terraform provider: https://github.com/eabrouwer3/terraform-provider-airbyte

Thanks for the feedback and we (Airbyte) are actively working on IaC solutions. We are publishing a Public API in 2023 [0]. We are working on an official Terraform provider built on that API, as well as, language specific SDKs. We hope to have these tools ready in 2023 as well.

[0] - https://api.airbyte.com (There is a roadmap there).

Totally agree - in fact from the issues pages it seemed like there had been a conscious decision from Airbyte not to support a Terraform provider and instead build out Octavia, which as you say leaves a lot to be desired. Deploying Airbyte is still far too hard, I'd be happy to pay for an 'on prem' version like Gitlab where it's a bit more managed. I also think the failure modes for Airbyte on prem are too hard to debug in comparison to other data 'MDS' tools.

Are you sticking with it or using something else?

Were you self hosting or using cloud?

We're considering this vs. Fivetran.

Hah. As if anyone listens to the concerns of Platform engineers. Product and data teams make their choices and we are left to pick up the pieces and glue them together into something passably functional.

We are self hosting. Don't think the cloud hosted version is being considered anymore now that I got something up and running.

Switched to Fivetran after experiences with Airbyte, operating Airbyte was extremely tedious, even primetime connectors had bugs that I personally had to fix, we keep Airbyte around for the "long tail" of SaaS tools that Fivetran doesn't have a connector for but managing an Airbyte deployment in a stable manner for the long term was not easy.

Reply on news.ycombinator.com