Netdata: Open-source real-time monitoring platform
github.com
github.com
A sever admin wants "Incident" panel that only shows anomaly components at the top coupled with adjustable alerting mechanism and not just a dump of all the data there is blindly.
There are so many tools that does this and pretend it's impressive including ELK but whether it's Grafana or Kibana, you need a lot of manual tweaking to make the dashboards actually useful.
We want our users to get 80% of the value with 20% of the effort.
It's an opinionated approach that liberates a lot of users from having to setup and maintain everything.
The alarms in netdata resolved a long-standing network issue on one of my boxes, and have variously alerted me to problems I could resolve with storage which greatly improved performance on my largest volume. On my other box, one look at the graphs alerted me to the fact that the entire SSD for my bcache volume was going unused[1]. I then used them while altering configuration and working with the drive to ensure the cache was being filled in a manner consistent with what the volume stored/how it was used.
The more I think about it, I might not have been as enthusiastic in my original comment as I should have been. It's been very helpful to me. I don't usually keep things like this running for very long (it wastes cycles on aging hardware...that isn't heavily used, but hey, it's the principal!) but I've kept this around because every time I've thought about removing it, I've visited the dashboard one last time and found something there that made me keep it.
[0] Though, as I mentioned, I'm not a sysadmin; I have a lab that might indicate otherwise, but I don't get paid for it.
[1] I had reloaded the machine/redone a previous configuration that included bcache and it screamed; I knew my new setup was much slower but I had forgotten about it until netdata made it obvious, again. I can't remember what I had to do to fix it, but it had something to do with the policy used to determine if a file should be put into the cache, and I think it was related to the fact that the cache was added to a volume with data present that rarely changed.
This is one of the things i am focusing on most - how to package and then surface up "anomaly events" to the user that the user can then quickly digest and decide if they are or are not something that could represent an "incident". So human in the loop sort of ML to help assist and lower the cognitive load of all the charts.
We have a first step on this ladder via the python based anomalies collector that you could play around with to see if any use: https://learn.netdata.cloud/docs/agent/collectors/python.d.p...
That will give two summary aggregation type charts based on anomaly scores built off all the system.* charts by default (but can be configured however you want) - so, at each second, an anomaly probability for each chart and an anomaly flag if the model thinks that chart looked anomalous at that time.
We are also working on some other related projects to build on this capability out more and do it at the edge, in C++ or Go as opposed to Python, (or via a parent node) as cheap as possible so minimal, ideally negligible, impact on the agent itself. We should have some more features related to this in coming months as we are just trying to dogfood them internally a little first.
Anyone interested i'd love to hear some more feedback here or in this community megathread here: https://community.netdata.cloud/t/anomalies-collector-feedba... or just email me at analytics-ml-team@netdata.cloud
What would you need it for? Once you defined your SLOs, either your service meets them or not. What's the value in alerting someone that "this graph looks funny"?
Where automatic anomaly detection was very valuable for us was in the video domain with multi dimensional end user telemetry. I.e what would be lost as noise in top level metics could be surfaced via anomaly detection for specific combination of dimensions that you could not otherwise manually observe. I.e video start time in Mexico is fine ... But an ISP in Mexico City is not failing but when data is sliced and anomaly highlighted we see its newly under preforming and we need to feed this data into our CDN switching to improve video start time there.
The data had too many dimensions that were always changing with degraded experience easily lost in the noise when measuring across platform and our software updates, combinations target devices, connection types, geo location, specific content, active ab tests, etc. In such cases automatic anomaly detection was pretty critical.
So doing it in a way where you can quickly scan such events i think could be useful even if only 1 in 20 actually turn out to be some potential problem that might have been missed by your alarms or maybe could even be a precursor to some impact on SLO's etc.
The aim would also be "this collection of graphs look funny at the same time" as opposed to "the individual graph looks funny" as if you have an anomaly score for every chart for sure at any given moment some individual charts would be randomly firing. But when you pool the information across charts and hosts and systems the hope is that then you can use anomaly detection as another way to explore your system and catch when things change unexpectedly.
What's the value in mentioning to someone that the chicken they're about to eat looks a bit raw?
It stops them from eating it and getting food poisoning.
Anomalies are often warnings (harbingers?) of a problem which could lead to a fault and downtime.
I for one deeply enjoy interacting with my Netdata dashboard whenever I want to deep dive into my servers resources and behaviors. For me it fits a purpose and if I ever were to run a company that hosted things, I would want it and I would want to pay for it. I am a huge fan and a long time homelab user of Netdata.
When, really, what's more important is actionable events, correlation, duplicate suppression, escalating notifications, etc. Something like what "Netcool Omnibus" and other commercial software does. Isolate actionable problems and make sure somebody owns the problem.
But for reasons I don't understand, there isn't much in the open source world in that space.
> never really actionable ... only of interest for a short period ... you know where and when the problem is.
Here's the thing: when something isn't right with those boxes, that's become the first place I visit. Since I had some franken-boxes with a bunch of storage, it's often related to the array, or btrfs. When I hop in there I'll notice an alert or two, gooble it, alter something and never see it again. It helped me solve some network issues.
I don't know, short of it being a more busy process than I wish it were on my server (only a little and I'm running a few plug-ins on the one that I'm unhappy with), it's been helpful.
Netdata is about making our lives easier. If you need to tweak Netdata, please open a github issue to let us know. It is a bug. Netdata should provide the best possible dashboards and alerts out of the box. If it does not for you, we missed something and we need your help to fix it, so please open a github issue to let us know of your use case. We want Netdata to be installed and effectively used with zero configuration, even mid-crisis, so although tweaking is possible and we support plenty of it, it should not be required.
An "incident" is a way to organize people, an issue management tool for monitoring, a collaboration feature. Netdata's primary goal however, is about exploring and understanding our infrastructure. We are trying to be amazingly effective in this by providing unlimited high resolution metrics, real-time dashboards and battle tested alarms. In our roadmap we have many features that we believe will change the way we understand monitoring. We are changing even the most fundamental features of a chart.
Of course at the same time we are trying to improve collaboration. This is why Netdata.Cloud, our free-forever SaaS offering that complements the open-source agent to provide out of the box infrastructure-level monitoring along side several convenience features, organizes our infra in war-rooms. In these war-rooms we have added metrics correlation tools that can help us find the most relevant metrics for something that got our attention, an alarm, a spike or a dive on a chart.
For Netdata, the high level incident panel you are looking for, will be based on a mix of charts and alarms. And we hope it is going to be also fully automated, autodetected and provided with zero configuration and tweaking. Stay tuned. We are baking it...
How do you make or plan to make money?
https://public-001.gitsense.com/insights/github/repos?q=wind...
In the last 30 days, there were 2 frequent and 3 occasional contributors. I honestly thought frequent contributors would have been much higher, which leads me to believe the project is quite mature and they don't need a lot of people to work on netdata.
Based on Crunchbase, they've raised about 33 million so far and if the number of people required to maintain netbase is low (relatively speaking that is), I can see them not really needing to worry about making money and I'm guessing they are finding value in gathering data for ML.
yes, this is right
> if the number of people required to maintain netbase is low (relatively speaking that is)
The Netdata agent is a robust and mature product. We maintain it and we constantly improve it, but:
- most of our efforts go to Netdata.Cloud
- most of the action in the agent is in internal forks we have. For example, we are currently testing ML at the edge. This will eventually go into the agent, but is not there yet. Same with EBPF. We do a lot of work to streamline the process of providing the best EBPF experience out there.
> I can see them not really needing to worry about making money
We are going to make money on top of the free tier of Netdata.Cloud. We are currently building the free tier. In about a year from now we will start introducing new paid features to Netdata.Cloud. Whatever we will have released by then, will always be free.
> I'm guessing they are finding value in gathering data for ML
No, we are not gathering any data for any purpose. Our database is distributed. Your data are your data. We don't need them.
p.s. i am the only person working on ML at Netdata and i can confirm we don't gather any data for ML purposes, which is actually my biggest challenge right now :) - convincing people the ML can be useful without having lots of nice labeled data from real netdata users to be able to quantify that with typical metrics like accuracy etc. I'm hoping to introduce mainly unsupervised ML features into the product that don't rely on lots of labeled data and have thumbs up/down type feedback and we can then use that to figure out if new ML based features are working or being useful for users. So any models that would be trained would be trained on the host and live on the host as opposed to in Netdata Cloud somewhere.
Yeah I would have to imagine that it would be an issue. This is just my personal opinion, but I think there should be a way to provide anonymized data for building models for anomaly detection. Maybe an opt-in feature, as it would benefit everybody using netdata, but this is just my own personal opinion.
We believe that the world will greatly benefit by a monitoring solution that is massively battle tested, highly opinionated, incorporating all the knowledge and experience of the community for monitoring infrastructure, systems and applications. A solution that is installed in seconds, even mid-crisis and is immediately effective in identifying performance and stability issues.
The tricky part is to find a way to support this and sustain it indefinitely. We believe we nailed it!
So, we plan to avoid selling monitoring features. Our free offering will never have a limit on the number of nodes monitored, the number users using it, the number of metrics collected, analyzed and presented, no limit on the granularity of data, the number of war-rooms, of dashboards, the number of alarms configured, the notifications sent, etc. All these will always be free.
And no, we are not collecting any data for ML or any other purpose. The opposite actually: we plan to release ML at the edge, so that each server will learn its own behavior.
We plan to eventually sell increased convenience features, enforcement of compliance to business policies and enterprise specific integrations, all of them on top of the free offering.
Plus it's good fun to look at a big dashboard and pretend you're Homer Simpson at the Springfield Nuclear Power Plant.
But netdata also supports alerts by e-mail or HTTP-based (e.g. for Slack), so why not just turn on notifications and live from them?
An example, drawn from several major incidents in my career. You get an alert, you narrow it down to a process or machine, you evict the machine from your serving population to remediate the incident, but how do you keep it from recurring? The anomalous thing isn't apparent in your monitoring data, so it must be among the bazillion statistics that a running system exposes, but which you can't afford to collect and monitor on a per-host, per-container, per-process level of detail. That's when you want something exactly like netdata!
Netdata does this too, with a ton of thresholds already set up by default. The list of active alerts is at the top, with a badge and everything. Notifications use a hook system, so you can use whatever mechanism you like. Personally I get emails for medium level alerts, SMS for high and above, and wall posts/notifications on my primary machine for crits. It took some tuning to get the thresholds right for me, all perfectly easy to do.
I agree I would prefer to have the active warnings more visible than the graphs, but one click away really isn't bad.
I think it's likely one's take on whether these tools are useful is very dependent on the system architecture.
https://news.ycombinator.com/item?id=11388196
https://news.ycombinator.com/item?id=17773874
https://news.ycombinator.com/item?id=26886792
(For commentary, I'm not being snarky.)
Beware - by default it'll send home telemetry, and the web UI will try to "register" your instance with some kind of cloud. I find this super annoying, but it's possible to turn it off; just not well documented.
There's also a lot of plugins that scrape many kinds of logs, look at process data, etc. Again, might be useful, but for a home user it's much better to turn it all off.
My notes here have a write-up of how to run Netdata via Docker with example config files that disable the unwanted features here: https://jake.tl/notes/2019-11-09-cozy-server#netdata
Doesn’t work well for monitoring multiple servers though from what I can tell.
That sounds super interesting. Would you be interested to share the use-case in our forums? https://community.netdata.cloud
Ping me (@odyslam in the forum) if you need any help. We are always looking for awesome stories like that!
BUT it can be configured to push data to Prometheus or similar (it's called a "backend") and some other integrations like notifications can be done.
Super neat project, very easy to set up. I highly recommend it to anyone who does performance troubleshooting. Netdata put into a standard Linux system will detect a lot of different things, like firewalls, containers, a lot of software like databases, queuing systems, mail systems and provide additional data every second.
Works flawlessly on Proxmox clusters!
Further clarity on this would be appreciated.
[1] https://github.com/netdata/netdata/blob/master/docs/quicksta...
BUT the UI for this is just a dropdown for each of your monitored servers. I've found I actually want to export data to a more robust system so I can view patterns across machines, too.
We complement the Netdata agent with Netdata.Cloud, a free-forever SaaS offering that maintains all the principles of the Netdata agent, while providing infrastructure level monitoring and several additional convenience features.
In Netdata.Cloud, infrastructure is organized in war-rooms. On each war-room you will find the "Overview" page, that provides a fully automated dashboard, very similar to the one provided by the agent, in which every chart presented aggregates data from all servers in the war-room! Magic! Zero configuration! Fully automated!
Keep in mind that Netdata.Cloud is a thin convenience layer on top of the Netdata agent. We don't aggregate your data. Your data stay inside your servers. We only collect and store a few metadata (how many netdata agents you have, which metrics they collect, what alarms have been configured, when they triggered - but not the actual metric and log data of your systems).
Try it! You will be surprised!
Awesome! I see the free tier is indeed looking generous. Just hooked up a node and looks good - I like the calculate correlations on alerts thing in particular.
>Keep in mind that Netdata.Cloud is a thin convenience layer on top of the Netdata agent.
I see. Didn't know/understand that.
On the claim node page - could you perhaps add the kickstart bash script code too? I find myself needing them one after the other yet they're on different pages
At the moment it's based on a short window of data so the focus is more for short term changes around an area of interest you have already found.
Longer term it would be cool to be able to use an anomaly score on the metrics themselves (or if a lot of alarms happen to be going off) to automatically find such regions for you so its more like surfacing insights to you as opposed to you having to already know a window if time you are interested in.
I didn't get that from the website until just now. I was looking and looking for how much it would cost to subscribe for our 150 dev/stg/prod VMs -- usually that's the killer.
Netdata blows a big hole in that by transmitting your usage information off of your box without getting permission.
we use the data we gather in order to make smarter product decisions. We want to invest resources where it matters, so we need to know how our users use the product.
We are also very detailed on what we gather: https://learn.netdata.cloud/docs/agent/netdata-security
Lastly, we just changed our analytics engine, from google-analytics to a self-hosted posthog, which is an open-source product analytics platform
Additionally, what's actually unfair is that you proceed with this spying without the consent of the user. Being upfront about it is not obtaining consent: it's just informing the user you're about to violate their (lack of) consent.
You must obtain consent from the user first, before transmitting their information. Otherwise, your software is spyware. (Disclosing that you're going to spy on the user doesn't make you not-spyware.)
> we use the data we gather in order to make smarter product decisions.
Yes, you transmit the private data of the user for the express purpose of enriching yourself.
Opt-out is unethical: you must obtain opt-in consent first. The data you are transmitting does not belong to you.
You choose to willfully install Netdata. You have to read the docs where the opt-out telemetry is clearly explained, before you can self-host it too. If you care, you can disable it.
I honestly don’t understand HN. Multiple commenters deriding a free open-source project for having basic telemetry to understand feature usage.
I did not choose to willfully install Netdata - I don't use it because it is unethical spyware.
Telling someone "if you stay where you are, I am going to do $THING_REQUIRING_CONSENT to you in 20 minutes" is not obtaining consent if the person doesn't, say, leave the building. Being in the hospital is not a blanket consent to anything the doctor wants to do, for example.
To transmit a user's private data (their usage) to the app vendor is unethical unless the user has specifically indicated that they want that to happen. If they haven't (and simply installing the software is not that), transmitting it anyway is, at best insanely rude, and at worst actively malicious (like, for example, how Netlify's CLI used to transmit "I opt out of telemetry" events, before I got them to stop).
Calling nonconsensual spyware "basic telemetry" is a euphemism.
No, in this case, you willfully signed up for a surgery and decided to skip reading the T&C.
>> user has specifically indicated that they want that to happen
You did this by installing the software without opting-out.
You sound entitled and spoiled. And by incessantly accusing netdata of being spyware, it feels like you are not willing to have a constructive discussion.
We actually mask the ip address (https://github.com/netdata/dashboard/blob/master/src/domains...) so it's not even sent - we just send "127.0.0.1" as the IP into our self hosted PostHog. Likewise with any URL, referrer type event properties that could leak a hostname to us - we don't want that data at all so explicitly mask it before even capturing it in our telemetry system.
Previously, when using a fairly standard Google Analytics implementation, we could not really have this level of control all that easily.
So the hope is that with PostHog we can do better here while still enabling some really useful product telemetry to help us understand how to make the product better over time and try catch bugs and issues quicker too.
Oh and we have removed Google Tag Manager (GTM) from the agent dashboard too so that that's no longer around as a possibility for loading other third party tags too.
You can read more here: https://github.com/netdata/netdata/blob/master/docs/anonymou...
p.s. PostHog is really cool - check it out: https://posthog.com/docs#philosophy, https://github.com/PostHog/posthog
I do appreciate the opt-in vs opt-out argument and I think on balance if opt-out helps us make this free product better over time and help our users then, so long as there is a clear route for people to opt-out, it's worth it and crucially important.
But this is indeed more like an opinion that individuals might differ on in terms of the pros and cons.
I personally love sending telemetry especially to help make the products I love better :) feels like im giving something back. But that is just my own opinion.
posthog or not, your target market is more sensitive to this telemetry crap than GP.
Here are a couple of thoughts that have guided us. Thank you for engaging in this conversation and for caring enough.
1) This data is crucial for us. We need as much as we can get and it's highly specialized to Netdata. (e.g a sudden increase in crashes will prompt our team to see recent changes) 2) The more friction we add (opt-in), the less people will do it (because people choose the easier route, always) and thus we will have less data to work with. 3) People who care enough, as you said, about their privacy, can *very* easily disable the anonymous statistics, by both adding a flag to the install script or doing a small config change afterwards. I feel that we are communicating in many different places that we take anonymous data, so most of our users should be informed. 4) It's a fairly standard industry tactic and I don't believe that other solutions are not doing. Of course this is not an excuse for anything, just noting that we are not an outlier.
Thanks again for engaging. Feedback is great for us, it makes us both happy (because someone cares enough) and better.
Settled on neofetch [1] instead: pure bash, wrote my own custom inputs including color coding for incident reporting in less time than it took me to strip down netdata. Highly recommended if you want to spend your time on other things than (setting up) server monitoring.
There's a reason timeseries databases are trying to get downsampling right.
Long term retention is now possible, but with relatively high memory requirements, depending on how many metrics are collected. Again, it was a decision to never give up realtime granularity and speed, even at the cost of writing our own timeseries db in C and utilizing more memory.
The centralized pull architecture of Prometheus does not lend itself towards small interval updates or towards resiliency (you actually need to run 2 Prometheus and double scrape for that).
I still install netdata on every machine though. Almost never use it, but there have been some times where it was useful to look at netdata. It's light weight enough that it hasn't been a problem.
The data are streamed from the Agent directly to your browser via the cloud.
Relevant docs: https://learn.netdata.cloud/docs/configure/secure-nodes#disa...
I wouldn't formulate it that way, it's just a bit annoying for me to see this trend of not having even tiny bit of security built in and having to do extra work just to protect the dashboard. Just one admin account and a random generated password would be fine.
stop being obtuse
How many times do we need to repeat this mistake?
If setting up a reverse proxy behind whatever monitoring you've got is too much, then yes, by all means use the SaaS offering -- but that's 100% the user responsibility, and there's no need to be snarky about it.
Are you speaking about Netdata or in general? Because if the former, then at least the updates part is not true: the installation script turns out nightly updates (and telemetry).
Frankly, the reason there is no basic auth is that Netdata doesn't use a third-party web server but a built-in one, so they would have to add this functionality.
More convenient because you can use your companies pre-existing authentication to authenticate the requests, and more secure because you're not having to manage separate passwords and user accounts.
We have a whole bunch of metrics that we keep track and we are currently implementing a load more.
Soonish, we will greatly increase the number of metrics that we gather with ebpf. That coupled with our per-second granularity, should give you a very detailed view of the system.
Docs: https://learn.netdata.cloud/docs/agent/collectors/ebpf.plugi... Community Forums discussion: https://community.netdata.cloud/t/linux-kernel-insights-with...