Hacker News Reader

Comment by torkelo | Hacker News Reader

Hi, my name is Torkel Ödegaard.

Grafana project creator & lead. Here to answer any questions.

Just wanted to say thanks for an awesome piece of software. It's one of those "I can't believe it's doing it and doing it this well, and it's free" kind of projects. For me, Grafana together with ElasticSearch and InfluxDB+Telegraf is pure monitoring/analysis/alerting bliss. Stay true to your roots and don't get tempted to turn into Kibana ;)

Thanks for leading the Grafana project!

As a color blind user, I find it difficult to read the dashboards created by others. How can I help any efforts to increase Grafana's accessibility?

I'm also colorblind and a heavy Grafana user but I haven't had much difficulty reading dashboards. What are you having trouble with? The most trouble I've had is colors of multiple lines on a chart being too similar but being able to use the mouse to select a line and see what the line is and its value helps there.

The default palette is definitely a challenge, since it has pretty low contrast and Grafana's defaults use a lot of thin lines.

I especially have trouble matching colors in a legend with the drawn objects in the graph. The best help has been the hover panel, but it's unsorted by default and the highlight contrast on the panel for hovering over a line is pretty dim.

Upvote: https://github.com/grafana/grafana/issues/18041 and add any additional comments and context there. In addition to making the light/dark themes more accessible, I think we should add an additional high contrast one.

Hi Torkel,

First off, Grafana is amazing! So keep up the good work.

Question: Currently alerts are fairly limited both in applicable panels and how they export/share relevant data to the chosen notification channel, as well as more advanced conditions. Is there any work going on with improving alerts?

Also to mention frequently asked feature: alerts using queries and templated variables [1][2]. Creating even simple generic dashboards with alerts is not possible, unless variables are hardcoded in hidden queries.

[1] https://community.grafana.com/t/template-variables-are-not-s...

[2] https://github.com/grafana/grafana/issues/6557

yes, this is a huge pain point for us as well

Agreed. If Grafana alerts supported templated variables I would migrate a decent chunk of my monitoring stack. But without templated variables the alert system feels half baked and I almost prefer it not even be a feature.

How about adding support for configuring Grafana with files?

This would allow us to deploy immutable Grafana instances using infrastructure-as-code tools like Terraform and Docker. Specifically, I want a way to eliminate post-install steps such as adding users and organizations and setting user passwords.

Wouldn't a better way to configure users and passwords be to use an external auth provider such as OAuth or LDAP? If you don't already have a auth provider, you could deploy Keycloak as your user DB.

OAuth/LDAP are great for large organizations. Small organizations don't need such complexity.

Even if you use OAuth/LDAP for accounts and passwords, Grafana still requires manual steps to create the organization, install plugins, and set the admin password. I wrote a Python script to do this on Docker container startup. That script is a workaround for Grafana server's poor ops design. It's extra code that we must maintain.

The admin password can be set with an environment variable. That just leaves the org and plugins.

At work we have to compile everything from source because compliance, and Grafana is one of the harder ones to get right. There used to be good documentation for it a few versions back but I can't seem to find it lately. Is there any good documentation on how to get Grafana compiled from source?

Checkout the contribute folder on GitHub. Here is the developer guide:

https://github.com/grafana/grafana/blob/master/contribute/de...

I just built it from sources because of the same reason. What are the issues on your side?

I've got a complicated proxy setup to contend with and have to build via Jenkins on some older Linux boxes, so it's just trial and error. You have a build script you'd be willing to share?

We are building from source with Docker. Without using Docker, you could try to just run the commands in https://github.com/grafana/grafana/blob/master/Dockerfile

Please let us know and/or open an GitHub issue describing if there are any certain part/step making it hard to build from source. Thanks

Zero experience with Grafana here.

I have a mongodb database with several collections. All collections share a set of fields: "createdAt", "channelId", "status", etc.

How easy would it be to dump all this data into Grafana, and then be able to slice and dice these collections by any combination of the shared fields.

Basically, I'm tired of writing mongo queries whenever I want to inspect my data. I want to point and click around a beautiful UI, with data visualizations and easy filtering/aggregation. Is this easy to do with Grafana?

I don't believe grafana will help you here.

If you have mongodb enterprise you can use Mongo bi connector and connect it with any bi tool. I recommend metabase since it is open source and very good for slice and dice and reports.

The mongodb bi connector is a postgres with foreign data wrapper to mongodb. So even if you don't have a mongodb subscription, you may be able to pull this off using the OS version https://github.com/EnterpriseDB/mongo_fdw

Grafana is a UI, not a database (it doesn't store any data.)

Plugins are used for showing pie charts, etc.

I've never tried this one, but the mongo datasource plugin exists: https://github.com/JamesOsgood/mongodb-grafana

Maybe you can use it directly this way? (if you don't mind the BI-equivalent hammering your primary data source)

have you seen any implementations of internal tracing like APM being used with Grafana? I think lack of a good open source alternative to New Relic or Datadog APM kinda killed OSS monitoring for my company.

The Jaeger integration is a good step in the right direction though, kudos!

What more would you like to see than what Jaeger provides?

Jaeger is used for traces between distributed systems (external)

Traditional APM products are internal traces

AFAIK, You can have internal trace markers, not just between apps. I've used that (for Jaegger) in one of our Windows Application to see the call-graph (it's a level editor for our game), and it was tracking the loading of various resources across multiple threads, then it was visualized with jaeger and even zipkin.

when you post release announcements that possibly get posted snd forwarded to other places, please include a short description about what grafana is.

and maybe also add one to your home page. (make it link to the features)

when i see any release anouncement about a product i am unfamiliar with, the first question i have is what kind of product it is, so i can decide if i want to check it out

Hi Torkel, this issue has been open since 2016[1]. There is incredible demand for it yet complete silence from Grafana on committing to the effort. Can I get official word on this please?

[1] https://github.com/grafana/grafana/issues/6983

Have there been no efforts to add this with a PR? I mean Grafana is open source and if the project leads wants to focus on other things that's his choice.

Attempts were made: https://github.com/grafana/grafana/pull/14075

The reason is we want to move alert rules out of the dashboard. To be their own thing, have their own UI & workflow for editing and updating.

Once that is done, we plan to add the ability to link an alert rule to 1 or more dashboard panels.

I just wanted to say how much I appreciate your work on Grafana. It's an indispensable tool that I couldn't do my job without!

Thank you!

Will Grafana ever add direct support for rrd datasources?

Does this help?

https://github.com/doublemarket/grafana-rrd-server

Kinda, but not really.

It has no support for rrdcached flush, AVERAGE CF is hardcoded , and I cannot seem to get it to not connect lines through gaps where there were no measurements.

Example not related to rrd, but same problem different db: https://github.com/grafana/grafana/issues/2125.

Hi Torkel, Just finished watching the GrafanaCon 7.0 Release talk. There are some great Usability improvements and new functionality which will be definitely useful for my upcoming project.

Thanks for the great work!

Thanks!

Is it now possible to create a horizontal lined based on a single value (either a fixed or a computed number)?

For example when I'm tracking the power grid frequency, I would like to add two lines at 50.05 and 49.95 Hz, as well as one at 50 Hz (with different color).

Others have voiced their desire to display a horizontal line at the 95th percentile in a bandwidth usage graph.

Maybe the thresholds feature in Grafana would work for you?

Otherwise depending on the database, they have functions for drawing lines or you can do a hack like this one for Prometheus:

https://play.grafana.org/d/h1lglb7ik/threshold-hack?editPane...

Amazing work we have been experimenting with grafana at work and the experience is awesome! Thank you for the awesome project

Thanks for adding more support for logs in Grafana.

I'm not using AWS CloudWatch Logs because of the financial risk of a runaway server running up a huge AWS bill. I had difficulty getting log display and log search working well in Grafana 6.6.2 and InfluxDB. Do you plan to improve log support for other databases?

Thanks a lot for creating it and pushing it forward. It's my go-to tool for monitoring dashboards and also some simple interactive data dashboards.

Never had any issues with it (and the ones I had were promptly fixed by the devs and the community).

Are we any closer to having some kind of anomaly detection in grafana (when using prometheus)?

Just curious: what kind of anomaly detection are you looking for? Can you describe the kind of thing you'd want this system to detect?

We have all sorts of data flowing into Prometheus and charted in Grafana. Grafana lets me alert based on thresholds and that works fine for many things.

EXAMPLE 1:

Imagine a service. Load avg of 2 is unacceptable and means things will be slowing down. However, a heavy job runs every night and pushes load averages between 1.5 and 2.5 during a 30 minute window.. I wanted to set a load avg alert threshold @ 2.0 but can't since it would fire false alarms too often. So i end up setting it at 3, and end up getting a late alert when things get really bad.

Instead, an anomaly detection would look for a number of standard deviations away from a mean, and ideally account for seasonality (i.e. compare the data with same data last few weeks). Now, things will behave better. If loads hit 2 during the day, it'll fire. But during that 30 minute window at night, loads would need to exceed 3 before it fires.

EXAMPLE 2:

Let's say I'm tracking 30 metrics for a service. I understand the service well enough to set thresholds in 5 metrics. However, I'm sure a failure won't happen alone and the other 25 metrics may show some oddities and give me an early warning of "something wrong happening" .. Here, some kind of loose "anomaly alerts" could help as an early warning. If a certain metric throws too many false alerts, we find a better way to monitor it.

EXAMPLE 3:

Lets say you have a service that auto scales. There's no limit to how many requests the service handles. So I can't really thresholds on requests/5min. Here, some kind of "anomaly alert" would be useful to let the team know that something is going on that's causing a high level of requests. Again, accounting for seasonality because things tend to change on weekends.

At the end of the day, an anomaly alert is not some "magic solution" but just and extra tool in the toolkit to use when it makes sense. Datadog and AWS both have built in easy to use anomaly alerting. Would be nice for grafana/prometheus to have something similar.

Have you seen this talk/blog from last year's Monitorama about anomaly detection with Prometheus:

https://about.gitlab.com/blog/2019/07/23/anomaly-detection-u...

Yes, I saw this last year. Complicated and requires a lot of effort on the user's part compared to let's say CloudWatch anomaly alerting: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

Thank you for this amazing, helpful project!

Are there plans for letting power-users style their own custom panels using html, css and js using just the web interface? Meaning without having to develop a plugin.

This has been possible for ages. The built-in text panel has an HTML option (though you have to disable this setting to allow script tags: https://grafana.com/docs/grafana/latest/installation/configu...)

There are some external plugins that offer more advanced support:

- https://grafana.com/grafana/plugins/aidanmountford-html-pane... - https://grafana.com/grafana/plugins/ryantxu-ajax-panel

There are a few plugins that let you enter css/html/js directly -- see https://grafana.com/grafana/plugins/ryantxu-ajax-panel https://github.com/srclosson/grafana-div-panel and the text panel (if the server is configured with disable_sanitize_html = true)

this is powerful, but long term tends to lead to a maintenance nightmare

Do you plan to make debug logging useful? I had difficulty troubleshooting Grafana & InfluxDB auth problems and found that debug logging did not provide any extra detail.

Does the new plugin architecture allow me to provision plugins from a config file on startup? Like installing the Kubernetes App without `grafana-cli plugins install kubernetes-app`

You can provision plugins on startup with something like: https://grafana.com/docs/grafana/latest/installation/docker/...

Does that method support removing plugins?

Someone already replied but I should add that most if not all config parameters can be passed to the container through env variables.

Do you know if performance has improved for large dashboards?

since when? :)

A few things have been added that may help: 1. only visible data is queried and loaded 2. If you have multiple panels that use the same data, you can now just reuse the results (or a subset of the results) in a new panel. This will cut down on additional queries that may not be necessary

i see you still have plans to continue with uPlot testing :)

https://github.com/grafana/grafana/pull/21835

absolutely -- with everything else for 7.0 that did not make the cut. But would love to work more with your frankly amazing library :)

Hey! Thanks for grafana, and especially a HUGE thank you for the user usage metrics!

Now if only you could persuade kibana to do the same :-)

Hi, I used Grafana in my last company and absolutely loved it.

How do I convince my current company to get the hell off Datadog?

The cost aspect is one thing. Vendor lock-in another. Probably need to have a team that can lead the transition towards Grafana, educate and speak for it. Grafana has an enterprise plugin for datadog that potentially could ease the migration

We just moved everything from datadog to Grafana, Prometheus and Loki, mainly due to cost. Honestly though, I feel like we get more out of the Grafana stack. Its powerful but simple enough to keep it pleasant to work with.

Haha, my team is going the opposite direction: we're leaving prometheus + grafana to datadog, and we're loving it. having every AWS monitoring default dashboard work right out of the bat is a bliss..

How about don't? Datadog offers everything that grafana does plus alerting, as well as more and better integrations (databases, AWS, middleware, etc...).

Reply on news.ycombinator.com