Hacker News Reader

Comment by shapiro92 | Hacker News Reader

before I try yet another ETL tool. How does this work with datasets that do not come from 3rd party providers like salesforce etc? I have had to build ETL pipelines for highly customized datasets either row level based or xml with I would say tricky code as the nesting or flows were not so simple and a lot of data missing.

How would Meltano or the other mention tools handle this?

Example is EDIFACT or FHIR or BDT

View on HN

We're working on an SDK[0] for building taps that should make it much easier to build to the Singer spec with all of the features out of the box. In theory if you can write some python against whatever you're pulling data out of, then it can work within the Singer ecosystem and Meltano. It's nearly ready to go but we'd love feedback if you decide to test it out!

[0] https://gitlab.com/meltano/singer-sdk

okay fair enough. but I guess then it doesn't offer anything extra for us as we have already the ETL Platform self made and we add the process with just an extra file. thanks!

I can speak for singer taps which this is based on for the EL bits. Our postgres etl works quite well. Logical replication broke down quite quickly, but primary keys + updated_at keys are working very well for us. I can't speak about XML though.

Why did logical replication break down for you?

Great question—-I don’t really know! We were using stitch and couldn’t really get a good answer.

Reply on news.ycombinator.com