Hacker News Reader

Comment by diminish | Hacker News Reader

could anyone tldr what meltano does and where the name comes from? is it mELTano of ELT?

I can't find the issue, but originally it stood for different parts of the data lifecycle - Model, Extract, Load, Transform, Analyze, Notebook, Orchestrate. We don't really highlight that much anymore, but there's no denying the roots!

Meltano is a open source platform for automating ELT pipelines.

Is there any significant difference from the traditional ETL?

I could imagine a scenario where the target server has more capacity and thereby a better place to do the transforming steps.

The idea is lean towards doing no transformation before loading into your Target Database/Datawarehouse. Do all transformations after the data hits your target database. DEV time is expensive, storage and CPU is cheap now. DBs and DWs are blazing fast now.

You can still only pull the data that you need using Meltano, but you select things on a "Stream" / "Attributes" level. For source DBs this means something like Table and Field level is as granular as you get.

not clear at all what it is

Meltano is an open source platform for ELT (Extract, Load Transform) pipelines that leverages Singer for data integration and dbt for transformation. It may not be clear from the blog post (since it's not focused on introducing the product), but I hope the homepage would get that across: https://meltano.com/

Please let me (Meltano CEO) know how we could make that more obvious!

https://meltano.com/ right now says:

"ELT for the DataOps era

Meltano is open source, self-hosted, CLI-first, debuggable, and extensible."

That doesn't tell me what Meltano is.

I happen to recognise "ELT" as "Extract, load, transform" but I imagine a lot of people would have to look up the acronym.

I hadn't seen the term DataOps before, despite working in this area. DevOps for Data? But the definition of DevOps itself is still pretty vague!

I know how hard it is to explain what a product like this does - I'm definitely not there with my own https://datasette.io/ project yet - but what you've got right now could definitely do with some improvement.

Here's my attempt for Meltano, based on a very loose understanding of what it is:

"Meltano is an open-source tool for building automated pipelines for extracting and converting data. Pipelines are defined using version-controlled code, and the framework can be extended using plugins written in Python."

This really isn't very good, but it does fill in the questions that I had when I first visited https://meltano.com/

Thanks! This is really useful feedback. To date, Meltano has been geared towards data folks who are likely to understand these concepts and acronyms, but there's definitely a need to explain it for folks who aren't as familiar. We're going to be working on the website copy in the coming weeks to make it easier to understand with as little digging as possible. If you think of anything else please feel free to reach out to us.

First off congratulations!

I was also confused as to what Meltano actually is or does, but everyone here seems to be so excited and genuinely happy for you and that made me super curious. It must have taken me several minutes of roaming around the website and docs before I finally figured it out though!

The main problem, I believe, is that there is too much copy that describes how the product will make its users feel etc. and far too little that tells us about its technical capabilities and its use cases. That notwithstanding, the copy is beautifully written and it has a very nice and smooth flow.

Being a developer myself and have seen/operated similar systems before made it not too bad to connect the dots. But that could just as well be exactly what you expect from your target audience!

Hi ... Get some corridor feedback :)

ELT isn't explained, and probably a few other terms as well. It's just not clear what either the value or purpose is.

In my opinion, that is.

Yeah, ELT is really pretty simple it gets hyped up too much as being super innovative. DEV time is expensive so optimize for it instead of saving CPU cycles / disk space.

Instead of trying to save space in your destination app by filtering data and only pushing what you need into your Database/Datawarehouse (Snowflake, Redshift, Postgres, MSSQL, etc) throw all of your data from your sources into the Database. They are super fast and storage is cheap. The expensive thing now is DEV time so optimize for that.

anagram of Menalto? (Menlo/Palo Alto)?

Reply on news.ycombinator.com