Hacker News Reader

Comment by kfk | Hacker News Reader

UI based flows for integrations are as bad as UI based flows for ETL. For some reason, we managed to move past UI to code for ETL, but now we are taking the exact same path for integrations. Integrations are complex, you might have complex requirements, polling and pushing needs depending on the applications, fast lookups needs, legacy IT solutions that hold critical data. I don't think we are ready to go full UI in this space, I think good orchestration and good microservices are still a better choice.

View on HN

Yeah, this is almost the sentiment behind the snarky comment I made that got voted down.

The difference is I am not so pessimistic about those UI-based ETL systems because I did a deep dive on the architecture of those and even worked for a startup that built one. I've seen how the industry's fixation on raw performance has kept ETL systems using a relational model that is hell to program for complex jobs -- I pitched a next generation system to a very acquisitive leader in this space and got shot down because my system wasn't friendly to storage locality and SIMD instructions. (That startup had the right idea of passing 'JSON' documents between the lines and the boxes but never settled exactly what the algebra was for those things, how to close event pipelines in the end properly so you always get the right results, ...)

Those "integration builders" though strike me a way of replacing a 10 line Python script with incredible tedium and "you can't get from here to there" experiences. That leads me to say things like "When I hear the word 'integration' I reach for my gun."

Check out Make, n8n, workflow86

Agree that coded integrations are still very relevant, but the addressable surface for UI is pretty large now.

Is there any open source, embeddable alternatives or frameworks? I've got an ETL DSL that is fully serializable and I've been looking to create a UI tool for it.

Maybe Airbyte if I understand what you're asking correctly.

Ah right. Yeah, reverse-engineering Airbyte's DSL and just "borrowing" their UI generation may be the move.

I don't know of a solution for ETL DSL -> UI but https://github.com/lowdefy/lowdefy goes from a DSL to internal web apps. Perhaps some inspiration could be taken from there. Good luck!

Thats pretty fascinating stuff, thanks for the recommendation.

> UI based flows for integrations are as bad

Right but in the same breath we should mention all code is terrible.

Depends where those UIs are on. If you have some sort of agreement set in place where they won't update the interface, then this is very much possible.

If it is relying on an 3rd party platform that you can't control and not know when UI changes are coming, then it can be bad for criticality.

However, I fail to know how urgent criticality is in ETL space, especially when UI changes can be as simple as "theres a new button" or "margin on form changed". The key process features and UI related to it would rarely change.

How about catering to non-programmers?

There is a space for that for sure, but when things get serious, the programming bit is 10% of the capability, domain knowledge, data expertise and SQL become more relevant. Those skills are not fulfilled with UI tools. Sometimes in fact UI tools make the skill gap bigger because new hires might know SQL, but for sure don't know your expensive proprietary tool of choice.

Zapier could easily add a DSL that mirrors their UI. It's not the UI that makes it powerful, it's the integrations and workflow.

My approach to integration for ETL was a DSL.

UI is bad for ETL...huh? I hope you're not talking about Stitch/Fivetran, because they're a godsend in my experience.

Reply on news.ycombinator.com