andrewm4894

Hey, I'm trying to get this project off the ground and looking for feedback, users, and maybe even some contributors.

Idea is simple way to just get decent anomaly detection on metrics. Dagster does the orchestration on top of whatever database you use. You just define batches of metrics in SQL and Anomstack will then ingest, train, score and alert on your metrics for anomaly detection.

Bit more work to do but just wanted to put it out there.

Basically I've built versions of this in almost all my last jobs and it's silly and kills me each time. Main use case in my head is like for daily, hourly, business metrics etc. But it can be any metrics really, just not anything too near real-time. Also not sure about how well it would scale but I figure for a lot of use cases being able to just stand up a decent anomaly detection system on your metrics could actually be very useful.

Anyway would love to hear any feedback, ideas, or suggestions.

we are in process of getting the plugin signed at the moment: https://github.com/netdata/netdata-grafana-datasource-plugin

hey - I work on ML at Netdata (disclaimer).

We have a big PR open and under review at moment that brings in a lot more logs capabilities: https://github.com/netdata/netdata/pull/13291

We also have some specific logs collectors too - i think in here might be best place to look around at the moment, should take you to the logs part of the integrations section in our demo space (no login needed, sorry for the long horrible url, we adding this section to our docs soon but at moment only lives in the app)

https://app.netdata.cloud/spaces/netdata-demo/rooms/all-node...

actually sorry in this case its more like parent-child

https://learn.netdata.cloud/docs/architecture/deployment-str...

and just dont have to claim the nodes to Netdata Cloud if you don't want to.

hey - i work in Netdata on ML

Just to mention there is this doc too that also tries to explain various deployment strategies

e.g. stand alone: https://learn.netdata.cloud/docs/architecture/deployment-str...

mmm nachos

Hey - i work for Netdata on ML.

We have recently created enterprise self hosted options for bigger customers who can't use cloud etc. (prob not as relevant here)

For self hosted at a smaller scale then you can have your own parent with multiple children streaming to it.

This is an example demo node which is also a parent for some other demo nodes. None of these need to be claimed to or signed in to cloud:

https://sanfrancisco.my-netdata.io/

It uses the same actual dashboard as cloud so that we only have one dashboard to maintain so you get the cloud dashboard locally basically and the parent can then kind of act like its own little Netdata Cloud.

A handful of features not available this way since they depend on the metadata being stored in cloud as opposed to on a parent node but we are trying to bridge that gap where possible such that the metadata could actually live on a parent.

Cool! - we always looking for feedback, feel free to hop into our discord, forum, or GH discussions (links here: https://www.netdata.cloud/community/) to leave any feedback or ask any questions if you run into any issues.

(cheers for the mention here too - always nice to try get some feedback and discussion going on HN as its so candid :0 )

> So I turned to Netdata. A one liner on each server and we had super sexy and fast dashboard for each server. No birds eye view, but fine. I then spent maybe 3-4 days trying to figure out how to get alerting to work (just email, but fine) and get temperature readings (or something like that).

I work in Netdata on ML. Just wanted to mention that as of last release a parent node will show all children in the agent dashboard so if doing again as of today a parent netdata might have got you the birds eye view as a starting point https://github.com/netdata/netdata/releases/tag/v1.41.0#v141...

(of course we also have Netdata Cloud which would have probably worked too but maybe was not as built out 3 years ago as is now - but don't want to go into sales mode and get blasted :) )

Can you give some specifics if you have a sec?

(Not trying to be smart or anything, genuinely trying to get feedback as feedback from random users on HN is actually a really good and different audience than maybe we usually hear from so just trying to get a couple of bullet points maybe if you have a few minutes).

Disclaimer: I work at Netdata on ML.

I do believe that the FE part is not open source but a custom licence.

So agent is still all open source but the FE code is not is my understanding.

Anyway not at all an expert here or anything but will try ping someone internally to explain it to me or jump in here to do a better job than me.

Yep, works for parents regardless of having a cloud account or not.

https://sanfrancisco.my-netdata.io/

^ this is a demo node with 2 children, can see all the nodes easily now via the parent node dashboard.

(Disclaimer: I work at Netdata on ML)

Here is a post I wrote at work about how the anomaly detection system I am working on works and some discussion around design choices etc.

Oh meant to add, here is a blog post on the airflow medium discussing it a bit more.

https://medium.com/apache-airflow/painless-anomaly-detection...

I created an Airflow provider to make it easy to just bring your sql to define metrics and then have decent out of the box anomaly detection dags running and sending alerts via Airflow.

Just started it recently and have been dogfooding it in work, would love for others interested to get involved too if of any interest.

It's going to be very useful to know how all this stuff works so you can apply it or even just understand it, it's strengths and weaknesses. The is still a lot of stuff to happen and be done before any of it near enough useful and safe and reliable to be just let loose, if anything will be more and more need for "humans in the loop" as we maybe delegate some stuff to be a bit more ML assisted or even semi autonomous.

The original question is a bit "why bother with anything?" so a little hard to answer I think the way it was phrased.

Are you a LLM or something? :)

Love all his courses!

Have bookmarked it to sit down and properly read later :)

Can this be enabled on grafana cloud?

I found this article that sort of covers similar ideas really interesting too.

https://home.robusta.dev/blog/stop-using-cpu-limits/

Nice!

I literally shared a link to the part of the code that shows we don't capture and record the ip address in our telemetry. You are being quite disingenuous calling things "spyware".

I do appreciate the opt-in vs opt-out argument and I think on balance if opt-out helps us make this free product better over time and help our users then, so long as there is a clear route for people to opt-out, it's worth it and crucially important.

But this is indeed more like an opinion that individuals might differ on in terms of the pros and cons.

I personally love sending telemetry especially to help make the products I love better :) feels like im giving something back. But that is just my own opinion.

Here is a guide around how to develop your own python based collector.

https://learn.netdata.cloud/guides/python-collector

Good to hear metrics correlation might be useful to you, just as background you can get more info here: https://www.netdata.cloud/blog/netdata-cloud-metric-correlat...

At the moment it's based on a short window of data so the focus is more for short term changes around an area of interest you have already found.

Longer term it would be cool to be able to use an anomaly score on the metrics themselves (or if a lot of alarms happen to be going off) to automatically find such regions for you so its more like surfacing insights to you as opposed to you having to already know a window if time you are interested in.

oh cool that's a nice tool.

p.s. i am the only person working on ML at Netdata and i can confirm we don't gather any data for ML purposes, which is actually my biggest challenge right now :) - convincing people the ML can be useful without having lots of nice labeled data from real netdata users to be able to quantify that with typical metrics like accuracy etc. I'm hoping to introduce mainly unsupervised ML features into the product that don't rely on lots of labeled data and have thumbs up/down type feedback and we can then use that to figure out if new ML based features are working or being useful for users. So any models that would be trained would be trained on the host and live on the host as opposed to in Netdata Cloud somewhere.

i think it was just a random share by someone.

Disclaimer - i work for Netdata Cloud.

We actually mask the ip address (https://github.com/netdata/dashboard/blob/master/src/domains...) so it's not even sent - we just send "127.0.0.1" as the IP into our self hosted PostHog. Likewise with any URL, referrer type event properties that could leak a hostname to us - we don't want that data at all so explicitly mask it before even capturing it in our telemetry system.

Previously, when using a fairly standard Google Analytics implementation, we could not really have this level of control all that easily.

So the hope is that with PostHog we can do better here while still enabling some really useful product telemetry to help us understand how to make the product better over time and try catch bugs and issues quicker too.

Oh and we have removed Google Tag Manager (GTM) from the agent dashboard too so that that's no longer around as a possibility for loading other third party tags too.

You can read more here: https://github.com/netdata/netdata/blob/master/docs/anonymou...

p.s. PostHog is really cool - check it out: https://posthog.com/docs#philosophy, https://github.com/PostHog/posthog

I almost think of anomaly detection as a UI/UX type tool to help users navigate the data/systems. So use ML to find "interesting" or "novel" periods of time in your architecture (in the sense that the ML thinks they look novel based on some model), and then enable a user who is ultimately best placed to decide if it's actually of real interest to them or more like a false positive that they can just ignore and move on.

So doing it in a way where you can quickly scan such events i think could be useful even if only 1 in 20 actually turn out to be some potential problem that might have been missed by your alarms or maybe could even be a precursor to some impact on SLO's etc.

The aim would also be "this collection of graphs look funny at the same time" as opposed to "the individual graph looks funny" as if you have an anomaly score for every chart for sure at any given moment some individual charts would be randomly firing. But when you pool the information across charts and hosts and systems the hope is that then you can use anomaly detection as another way to explore your system and catch when things change unexpectedly.

Disclaimer - i work at Netdata Cloud on ML.

This is one of the things i am focusing on most - how to package and then surface up "anomaly events" to the user that the user can then quickly digest and decide if they are or are not something that could represent an "incident". So human in the loop sort of ML to help assist and lower the cognitive load of all the charts.

We have a first step on this ladder via the python based anomalies collector that you could play around with to see if any use: https://learn.netdata.cloud/docs/agent/collectors/python.d.p...

That will give two summary aggregation type charts based on anomaly scores built off all the system.* charts by default (but can be configured however you want) - so, at each second, an anomaly probability for each chart and an anomaly flag if the model thinks that chart looked anomalous at that time.

We are also working on some other related projects to build on this capability out more and do it at the edge, in C++ or Go as opposed to Python, (or via a parent node) as cheap as possible so minimal, ideally negligible, impact on the agent itself. We should have some more features related to this in coming months as we are just trying to dogfood them internally a little first.

Anyone interested i'd love to hear some more feedback here or in this community megathread here: https://community.netdata.cloud/t/anomalies-collector-feedba... or just email me at analytics-ml-team@netdata.cloud