Show HN: Off-premises insight for on-premises Prometheus servers | Hacker News Reader

Show HN: Off-premises insight for on-premises Prometheus servers

latency.at

It is interesting to see this because (AFAIK, etc.) the only option for Prometheus has been on-premises. I had hoped to batch analytics and import into Prometheus out of production (aka off-premises), but my very preliminary investigation appeared to show that it requires "live" data, since it assigns the timestamp when events are ingested.

I'm still confused whether or not bulk import will ever be a supported scenario.

https://github.com/prometheus/prometheus/issues/398 Improve staleness handling (2014-2017)

https://github.com/prometheus/pushgateway/pull/119 Reject pushes with timestamps (2017)

Bulk imports are planned: https://github.com/prometheus/prometheus/issues/535

Note that bulk really means bulk. This is for one-off loads, not doing push.

Any thoughts on a timeframe? This is the only thing holding me back from using Prometheus in any sort of serious way.

There's no hard timeline as we're an open source project, but hopefully in the next 6-12 months.

If this is holding you back from seriously Prometheus, then Prometheus is possibly not the right tool for you. Prometheus is about the here and now.

Despite your consistently dismissive attitude, it IS the right tool for me, the teams I work on/with, and side projects.

If that's not what prometheus is for, you should seriously consider adding a more-explicit mention of it in the documentation. Going through the docs, I'm finding nothing plainly obvious mentioning that about prometheus. At least put it in the FAQ, ffs.

What needs to be code-wise to get it going sooner?

I know that https://cloud.weave.works somehow includes or uses hosted prometheus. Haven't tried it though.

The 2.0 release of prometheus will help with stale data but not support bulk import yet. For that you can track https://github.com/prometheus/prometheus/issues/535

But to be clear: My service isn't about running Prometheus off-premises. It only provides the metrics which you can gather with your own Prometheus servers.

If you want not on premise, datadog and signalfx are very very good for monitoring/metrics.

Plug: I have an open source project that does something similar:

https://github.com/chrissnell/crabby

It allows you to run offsite HTTP[S] and page load tests and send the metrics back to a central collection system. It supports Prometheus (albeit through a pushgateway) among other metrics engines.

I was just brainstorming this type of use-case for Tectonic[1], which includes Prometheus as a cloud service that runs on-prem and on cloud providers. Remote monitoring is just as important as app metrics.

Very cool project!

[1] https://coreos.com/tectonic/

Thanks!

I just realize there is an open issue on Prometheus which, would it get implemented, would allow for very nice integration of the blackbox exporter as well as this service into kubernetes and tectonic: https://github.com/prometheus/prometheus/issues/3071

so basically blackboxexporter as a service?

Yes, basically geographically distributed blackbox exporters as a service.

I've added the necessary account functionality and also extended the http probe to split response times by 'phase' (dns resolution, tls handshake etc).

The later part will be upstreamed soon though. Already discussed the required changes with the blackbox-exporter maintainer.

It's a pretty nice service that shows the power and extensibility of Prometheus.

Reply on news.ycombinator.com