Comment by stochastimus | Hacker News Reader

Comment by stochastimus | Hacker News Reader

Hey folks,

Larry, Ajay and Rod here!

We're excited to share Zebrium's autonomous incident detection software. Zebrium uses unsupervised machine learning to detect software incidents and show you root cause. It's built to catch even "Unknown Unknowns" (problems you don't have alert rules built for), the FIRST time you hit them. We believe autonomous incident detection is a important tool for defeating complexity and crushing resolution time.

* Get Started *

1) Go to our website and click "Get Started Free". Enter your name, email and set a password. 2) Install our collectors from a list of supported platforms. For K8s it's a one command install. Join the newly created private Slack channel for alerts (or add a webhook for your own) 3) That's it. Automatic incident detection starts within an hour and quickly gets good. You can drill down into logs & metrics for more context if needed.

Getting started takes less than 2 minutes. It's free for 30 days with larger limits and then free forever for up to 500MB/day.

* Here's what you WON'T have to do *

Manual training, code changes, connectors, parsers, configuration, waiting, hunting, searching, alert rules, etc! It works with any app or stack.

* How It Works *

We structure all the logs and metrics we collect in-line at ingest, leverage this structure to find normal and anomalous patterns of activity, then use a point process model to identify unusually correlated anomalies (across different data streams) to auto-detect incidents and find the relevant root-cause indicators. Experience with over a thousand real-world incidents across over a hundred stacks has confirmed that software behaves in certain fundamental ways when it breaks.

It turns out that we can detect important incidents automatically, with a root-cause indicator when the logs and metrics reflect it. Zebrium works well enough that our own team relies on it, and we believe you'll want to use it, too.

Do you already have data models and ML to handle these tasks today, or are you still building data models with the aid of the clients who sign up for the service?

If you have a functioning platform today, what percentage of events end up needing human escalation?

Asking these questions considering the early stage of your product.

Hi, one of the founders here: The service is designed to not require specific data models, because that does not scale, nor does it keep up with changes in application behavior. Instead, the ML engine learns data structures, normal behavior of logs and metrics, and normal correlations between them for each app deployment on the fly. Then when things break it does a very good job of generating incidents. We make user feedback easy, so if we are "over-eager" in detecting a certain kind of incident, your response trains the ML quickly. We do improve the ML engine with experience of course (and have added some user controls), but now have dozens of applications using us, and cumulatively have over a thousand successfully detected incidents under our belts.

Hi eganist,

The tasks the SW handles are (1) detecting when it looks like an incident should be raised, (2) gathering up all the evidence around that incident, and (3) notifying the user via Slack or other webhook. The purposes are to (a) detect unknown unknowns for which you won't have an alert rule built, and (b) reduce MTTR by having pulled together evidence of impact and root-cause into an incident report.

Re: human involvement: a human still needs to review and potentially act on the incident report. The idea is that we've (1) alerted on an incident and (b) given you a great summary / starting point. You can provide feedback on the report to let the system know which sorts of incidents were good / ok / lame, and the system will refine future incidents based on this simple feedback mechanism.

Re data/data models: there are no app-specific/ stack-specific / user-specific rules built-in, and each user's dataset is learned/structured independently. As an example, suppose you are running postgres. We will learn that logstream structure from scratch as it comes in; there is no code that looks for the word "postgres" or "replication", for example, nor is there a built-in understanding of the timestamp formats supported by postgres. The structuring is done de novo for each log stream, anomalies are detected in very generic ways, and the model that decides what rises to the level of an incident is abstract and works the same for any app, also without special rules.

This design is quite intentional: there won't be any pre-built rules or connectors for your application, for example, and autonomous incident detection only works if it can grok an arbitrary stack OOTB.

Here's a link to a blog that shows how the system works on a few sorts of incidents, although the UI is much prettier now with charts instead of just text in the incident report:

https://www.zebrium.com/blog/is-autonomous-monitoring-the-an...

I hope this has answered your questions!

Hey.

What does unsupervised learning mean in detail?

Is this a deep learning or a classical machine learning approach?

There are three components to Ze that involve what you'd think of as ML. We use a few different "classical" techniques together; our focus is on keeping costs down while providing useful results on day one on even a newly-deployed custom application.

First is the structuring of logs: we have a four stage pipeline for structuring and each stage has greater importance depending on how many ground truth instances there are of a given event type in the dataset (these are unlabeled, of course). These stages include heuristics, reachability clustering, a naive Bayes classifier with global fitness function, and a modified LCS. When the data comes in it is laid down in tables with typed columns directly, without post-processing; later, table merges are considered asynchronously. This lets you start doing anomaly detection really well right away.

Next is anomaly detection. There are lots of dimensions we consider for AD; at the end of the day most of them boil down to some reflection of either “badness" or "rareness". We'd rather catch too many anomalies than miss one. This AD is run on both logs and metrics.

Finally, there's incident detection: here we look primarily at how much independence is there between different streams w.r.t. their anomalies, and when we see an unusually high correlation across channels, we raise an incident. Here, a naive Bayes model is used to set cutoffs, with streams of anomalies considered as point processes. You can provide feedback here for training, but it is optional.

Thanks for the answer, I really prefer clear statements about the methods being used over marketing-like loose labels like "ML".

Reply on news.ycombinator.com