Show HN: Using LLMs and Embeddings to classify application errors

Hi Hacker News! We’re Vadim and Chris from Highlight.io [1]. We do web app monitoring and recently worked on using LLMs/embeddings to improve our error product. Fair warning that we don’t have too much experience with LLMs, but since we’re seeing some interesting results, we figured we’d share our learnings with the HN community.

Our goal was to build two features: (1) tagging errors (e.g. deeming an error as “authentication error” or a “database error”); and (2) grouping similar errors together (e.g. two errors that have a different stacktrace and body, but are semantically not very different).

Each of these rely heavily on comparing text across our application. After some experimentation with the OpenAI embeddings API [3], we went ahead and hosted a private model instance of thenlper/gte-large (an open-source MIT licensed model), which is a 1024-dimension model running on a Nvidia Tesla T4 on Hugging face [4].

Our general approach for classifying/comparing text is as follows. As each set of tokens (i.e a string) comes in, our backend makes a request to an inference endpoint and receives a 1024-dimension float vector as a response (see the code here [5]). We then store that vector using pgvector [6]. To compare any two sets for similarity, we simply look at the Euclidian distance between their respective embeddings using the ivfflat index implemented by pgvector (example code here [7]).

To tag errors, we assign an error its most relevant tag from a predetermined set decided by us. For example, if we tag an error as an "authentication error" or a "database error", we can allow developers to have a starting point before inspecting an issue (see the logic here [8]).

Anecdotally, this approach seems to work very well. For example, here are two authentication errors that got tagged as “Authentication Error”:

* Firebase: A network AuthError has occurred

* Error retrieving user from firebase api for email verification: cannot find user from uid.

We also use these error embeddings to group similar errors. To decide whether an error joins a group or starts a new one, we decide on a distance threshold (using the euclidean distance) ahead of time [9]. An interesting thing about this approach, compared to using a text-based heuristic, is that two errors with different stack traces can still be grouped together. Here’s an example:

* github.com/highlight-run/highlight/backend/worker.(*Worker).ReportStripeUsage

* github.com/highlight-run/highlight/backend/private-graph/graph.(*Resolver).GetSlackChannelsFromSlack.func1

Both reported as `integration api error` as they involve the Stripe and Slack integrations respectively. The neat thing is that the LLM can use the full context of an error and match based on the most relevant details about the error.

We haven’t rolled this out to users yet, but feel free to give the logic a try at [2]. Long-term, we have lots of other ideas of what we could build with LLM tooling in observability, and if the HN community has any ideas, we’re all ears. Let us know what you think!

Links:

[1] https://news.ycombinator.com/item?id=36774611

[2] https://app.highlight.io/error-tags

[3] https://platform.openai.com/docs/guides/embeddings

[4] https://huggingface.co/thenlper/gte-large

[5] https://github.com/highlight/highlight/blob/main/backend/emb...

[6] https://github.com/highlight/highlight/blob/main/backend/mod...

[7] https://github.com/highlight/highlight/blob/main/backend/pub...

[8] https://github.com/highlight/highlight/blob/main/backend/pri...

[9] https://github.com/highlight/highlight/blob/main/backend/pri...