Comment by shinryudbz | Hacker News Reader

Comment by shinryudbz | Hacker News Reader

Parent Full thread

Hi everyone, I'm one of the founders of Overseer. We built this tool because we noticed that engineers had to dig through many dashboards when diagnosing an incident. We felt this process could be streamlined through the use of machine learning.

When we first started working on this project over a year ago, we weren't sure if the algorithms would work, or if our insights would be of value to anyone. We were also struggling to figure out how to make it easier for people to try the product without having to change their existing workflow.

Since then, we've made huge improvements to the algorithms, deployed the tech for several large customers, and demonstrated value. Now I'd love to get a bit more feedback from you guys and see if we're going in the right direction!

So here's how the tool works: 1 - We pull down your dashboards from your existing monitoring tool (e.g. Datadog/Wavefront/Librato) using your API key. 2 - We integrate with your PagerDuty account via a Webhook to notify us when an incident has triggered. 3 - When our Webhook is invoked, we will use machine learning to rank your dashboards, rank the metrics on those dashboards, and notify you via Slack/Email of the top dashboards/top metrics on those dashboards to look at.

For this demo, we only expose the Wavefront plugin, and you'll be able to configure it on the initial page.

To integrate with PagerDuty, you'll need a URL to our end point, and we'll need an email address where we can send the analysis. You can configure that by clicking on your user name (on the top right) and doing the following: 1. Clicking on "Generate API Key" and jotting down the generated Webhook URL. PagerDuty will need that. 2. Filling out the "Organization Email" text box. We will send your our analysis there!

Given that we'll be dealing with potentially sensitive data, we reluctantly decided to add a layer of security and have folks register with us first - this allows us to protect your data better. My apologies for the inconvenience.

I'd love to see what the HN community thinks and how we can make it better!

The thing that comes to mind is that maybe the incident should create a notification that also generates its own dashboard specific to the components that are (likely) involved. So rather than scanning or hunting for the right collection of indicators, they're all in the notification (or a URL within it). Note: I'm pretty sure this doesn't exist in a turnkey form.

There are a couple things I think it's possible you're saying: (1) When an incident happens, you should get notified what components are likely involved. That is basically what we do if you hook us up to PagerDuty- when PagerDuty sends out an alert, we will also send out an email that lists what dashboards are likely relevant along with links to Overseer dashboards that show you which metrics are probably relevant. (2) By "the incident should create a notification" you might mean you'd like Overseer to generate an alert. While this would be possible to do, where Overseer really aims to shine is as a triaging tool rather than being the best alerting service.

It could also be streamlined by just understanding your key metrics and visualising those over spunking out chart porn.

Adding magic into the mix hoping to surface the right thing at the right time but not being entirely sure that would be the case seems like solving a problem of unknowns by hoping that adding new unknowns will cancel out the old ones.

Solid point! I agree that it's super important to know your system well enough to know what your key metrics/kpis are without us. We are no substitute for that. One of the key places we try to fit in is after you may already know something is going on (ex, your kpis are being affected) and you want to drill down deeper to understand what's going on at a more detailed level. With a deep enough understanding of your system, you may have some ideas about where to look for more info, but there can sometimes be so many places to look that it's hard to know where to start. That's where we come in. One of the articles that really influences how I personally think about this stuff is here, it's a great read if you have the time: https://medium.com/@copyconstruct/monitoring-in-the-time-of-... . In the words of the article, we fit in between the overview and debugging part of the monitoring pyramid.

In a perfect world, sure everyone would know right where to look when triaging an incident. That’s a largely impossible task in real life, at least in my experience with larger teams/systems.

Reply on news.ycombinator.com