Anyone working on LLM tools for enhancing data quality? | Hacker News Reader

Anyone working on LLM tools for enhancing data quality?

5 pointscstanley7 comments

Big problem, let's break it down...

1. Data issue identification 2. Solution and implementation

Most issues are discovered in the data warehouse. Entity matching customer data across different systems, some business process results in duplicate, or null data. I know there are existing, non-LLM, products that do this. I'm curious to compare those with new LLM first products.

On solution/implementation. Ideally you're able to fix this in the source system, either in your SaaS tool or in the way you write production data. You can also fix this in the data warehouse, munging/ETL'ing the data. Seems like LLMs could help to 1) identify and recommend a change in an external system, 2) submit a PR to solve this in the data warehouse.

Anyone know anyone working on these problems?

Obviously, the cost of using LLM to solve this problem is relatively high. At most, it can only treat LLM as one of the signals instead of processing all the information. Combining LLM with tree models or graph models and adding NER technology may achieve an approximate solution to 2. Can you really trust the answers provided by LLM to be used in a production environment?

The combination is truly key, how we do it:

- Combining LLMs, gradient boosting, and multiple statistical/heuristic based approaches - For LLMs: rely on (quite) large context length + fine-tuning - The (ML) models are used to extract mathematically provable quality checks

1 is already a solved problem. My employer had originally put a ML based system to find DQ issues (already working) and is looking at pocs to add LLMs in the model mix. Hearsay is that our lakehouse vendor will have their own solution to this question via a acquisition.

2 is interesting, possible to do via LLM but I worry about data privacy and hallucinations making data more believable but not real.

reg. 1) Which one has been used?

I'm using LLMs to help build dbt-flavored Great Expectations tests. Pretty low-hanging fruit, but it's a start.

We (https://dqc.ai) are doing something in the space, yes. Next to a mixture of ML, and heuristic based approaches, plus link & integrations into source systems. Happy to talk about it, feel free to reach out.

Reply on news.ycombinator.com