Prometheus and Kubernetes up and running | Hacker News Reader

Prometheus and Kubernetes up and running

coreos.com

85 pointsphilips5 comments

Anyone running coreOS in production want to comment on their stack? We tried using it around a year+ ago, but etcd/fleet were too unstable and stuff like btrfs and other major features were being changed too often to get the supposed benefits.

It looks significantly more mature now, but I'm still not sure about the buy-in on all their various products.

etcd has had significant improvements in lifecycle management and stability in the last year. You can learn a bit more about the upcoming etcd v3.0 release here: https://speakerdeck.com/philips/etcd-mission-critical-key-va...

For the most part we are recommending folks use Kubernetes for new projects over fleet. Kubernetes "completed our sentence" a few months after we started fleet and has rapidly matured. The kubernetes v1.3 release bring a number of new features as well: https://coreos.com/blog/kubernetes-v1.3-preview.html

And on your btrfs question we moved off of btrfs onto ext4 + overlayfs as soon as overlayfs became a viable option in late 2014. You can read more about that here: http://thread.gmane.org/gmane.linux.coreos.devel/1952

As for users and maturity some of the videos from Tectonic Summit Day #1 from this blog post might be useful: https://tectonic.com/blog/tectonic-summit-wrap-up.html

Happy to answer any other questions on IRC or email: https://coreos.com/community

I've deployed almost exactly this on a new kubernetes cluster running coreos exclusively with 4 nodes, 3 masters and 3 etcd instances divided over 4 physical machines.

Things I haven't implemented yet is deploying the node-exporters via DaemonSets and the Prometheus config through a ConfigMap. Those are right now done through a cloud-config systemd override and a gluster mount.

Couple of things I ran into. The kubelet's running the kubernetes master components need access to the ssl certificates of the apiserver, otherwise they cannot be scraped by prometheus over https when they communicate with the apiserver over https. And I'm still very confused about a seemingly simple thing of getting a per request response time query. This is what I'm using now:

`irate(request_processing_seconds_sum{app="myapp", method=~"$method"}[5m]) / irate(request_processing_seconds_count{app="myapp", method=~"$method"}[5m])`

Two things: I'm confused by what I should set the vector time to `[5m]`. And how I can get the response time of an individual request. We've had a request take 30 seconds but that spike only showed up when viewing the graph over a 3 hour time period. When viewing it over 6 hours or 1 hour it simply will not show up even though it happend in the last hour.

Things I do really love is the ability of configuring services to be scraped by setting annotations to it. Works great when slowly transitioning your services to prometheus style metrics!

If any body has some pointers about querying and visualizing using grafana and prometheus that would be great!

Is there a tutorial to run Prometheus on pure CoreOS cluster?

The container guide provided by Prometheus is a good start: https://prometheus.io/docs/introduction/install/

Pre-build containers are hosted on Quay.io: https://quay.io/repository/prometheus/prometheus

Reply on news.ycombinator.com