Hacker News Reader

Comment by glutamate | Hacker News Reader

Some people, when confronted with a complex system, think "I know, I'll use a large language model." Now they have two complex systems interacting.

View on HN

The human body is orders more complex than LLMs and we can already use LLMs to improve anomaly detection rates for cancer.

> improve anomaly detection rates for cancer.

What do you mean by that?

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11003452/

“We propose and demonstrate the use of these models to assist in the manual curation process which results in higher accuracy and F1 score with lesser time and cost, thus improving efforts of cancer research.”

How does this relate to anomaly detection?

Anomalys are markers to keep track of when diagnosing.

That paper doesn't even mention the word anomaly or detecting them.

Cancer is anomalous cell behavior.

> Now they have two complex systems interacting.

As opposed to sending crowds of people to interact with the system? We are even more complex and less benchmarked/tested than LLMs. At least the LLM is a known thing, we know its limitations, we can evaluate and compensate for their lacks.

Some people trivialise things in bad faith to score internet points on Hacker News, I guess. The status quo, per the article, was already using “deep learning” models to perform anomaly detection. We are already talking about complex, black-box systems.

The stated advantage was that they didn’t require deployment-specific training. You can just throw a pre-trained LLM at it. There is also a stated benefit: early-stage detection, without needing to pay for, or wait for, training a custom ML model. The article is quite open about the fact that the LLM approach doesn’t beat the state of the art in terms of accuracy.

It’s like you didn’t click the link.

>early-stage detection, without needing to pay for, or wait for, training a custom ML model

This doesn't make much sense though, if you're in a hurry to get something working and don't have the data for training a custom model, you'd use the knowledge you have of the system to setup a typical statistical anomaly detector. That way you'd have the insight into the system behavior that you'd want early on, would be able to enforce all sorts of guarantees and would not have the weight and complexity of an LLM on the system's resources, rather than worrying about this LLM that has nowhere near the same info on the system.

Using an LLM for this would be like hiring a random person off the street to keep your network infrastructure working because you're in such a rush that you don't even want to document how your infra is setup, let alone wait to find someone who would know how to decipher it.

The metrics OP suggest increase with the transition from DL to LLMs (and OP had no immediate duty to discuss other metrics).

Colour me old-fashioned, but i am really sceptical about a zero-shot general-purpose LLM-based anomaly detector being able to outperform a trivial, not-obviously-wrong ML/statistical model calibrated on a small subset of the data that makes training extremely fast, in entirely new domains. And in anomaly detection, false negatives (and even false positives) can be very expensive.

The most promising role for LLMs in timeseries prediction is in extracting covariates from unstructured data, in the wind example, weather reports and geographically proximal social media posts.

You are skeptical about something that is explained quite thoroughly in the article. Did you read the article?

Edit: I meant they never claim to outperform SOTA domain-specific approaches. So, I don't know whose claim you are refuting.

> While LLMs could not beat state-of-the-art deep learning models at anomaly detection, they did perform as well as some other AI approaches.

I think one should also be sceptical about the current hype train

Funny how one of the the few comments actually discussing the content of the article gets downvoted while generic memes that have nothing to do with it get upvoted.

Reply on news.ycombinator.com