Ask HN: No-code data import pipeline builder, possibly a good idea? | Hacker News Reader

Ask HN: No-code data import pipeline builder, possibly a good idea?

2 pointsChiff04 comments

I have an idea for a tool that would allow users to build data migration pipelines for their onboarding users from other (possibly competing) software. In my mind, it makes sense, since it would be great for UX, and, from my knowledge, data migrations are usually done manually and are just boring and complex. However, I haven’t really seen anyone doing that, at least successfully, so I wonder, why? My guess is that it’s really hard to to build a solution that could do something like that without much specific work on specific migrations, and the only way to find success is to basically be a consulting firm that does one-off integrations. Would love to hear your thoughts though.

Cheers!

View on HN

There is a certain viewpoint that ontology should be at the heart of "no code". If you think that you way you believe that the key part of applications programming is developing a mental model for the application domain and then constructing data structures, user interfaces and such around that model.

I've seen quite a few attempts at automatically generating mappings from one data model to another. In the age of tf-idf this was inevitably a disappointment. With something like sbert.net we can certainly do better.

I'd consider two scenarios: (i) migrating data from a production system to an analytics system, and (ii) migrating data from one production system to another.

Let's imagine the system is a hotel reservation system. In the case of (ii) any scrambling of the data could produce a very unhappy customer. You might test the migration multiple times but at some point you have to cut over the old system to the new system and live with the migration you did. In the case of (i) an error might throw off your analytics, but the analytics might help you find errors, and you can always feed back what you learn into the migration process and do it again. (In fact you'll probably 'migrate' again and again to make monthly, weekly reports.)

Thanks for the reply. If I understand correctly, it would be a safer option to focus on data migrations between production and analytics systems?

I think so.

I spent some time pitching ideas for an intelligent data analysis tool and I have slide decks for it kicking around. Later on I worked for a startup that developed such a tool. If you contact me through my profile I can send you some decks.

Ok. Sent a mail.

Reply on news.ycombinator.com