I think the issue is that many of these metrics (e.g. RAGAS) are LLM as a judge metrics. These are very far from reliable. Making them reliable is still a research problem. I've seen a couple of startups training their own LLM judge models to solve this problem. There are also some work to attempt to improve the reliability through sampling such as G-eval (https://github.com/nlpyang/geval).

One need to think of these metrics as a way to filter all the data to find potential issues, and not as a final evaluation criteria. The golden criteria should be human evaluators.