Not sure how to filter through say 300 servers and select out the top 10 for a particular time span. I would think that it would be a common need but I guess I'm missing something?
For example, instead of highestCurrent, you could do something like:
topk(3, my_metric)
This would at every point along the graph select the current top 3 series that have the my_metric metric name.
Or if you want to average each series over the last 10 minutes at every point in the graph before selecting the top 3:
topk(3, avg_over_time(my_metric[10m]))
Note that due to the reasons mentioned above, topk() here does not select whatever line has the largest area under the entire visible graph range, but whatever is at the top at each given resolution step. So you may actually get more than 3 series in your graph, but only 3 at a time at any given X.
There's also an issue asking about this, but we're not sure if that is fundamentally compatible with Prometheus's query execution model without major changes: https://github.com/prometheus/prometheus/issues/586
Still look forward to rolling out Prometheus for all of the other great features. Congrats on the release!
You can calculate the highest value now (topk) or the highest averages (topk+avg_over_time) and now that you know which timeseries you want, graph those. I believe this is doable in Grafana.