OpenCensus – Vendor-agnostic libraries to provide metrics collection and tracing | Hacker News Reader

OpenCensus – Vendor-agnostic libraries to provide metrics collection and tracing

opencensus.io

64 pointsmooreds20 comments

One of the contributors to OpenCensus here - feel free to ask any questions!

Why is open census so expensive? Using the default ocgrpc views, we've exported the data to stack driver and datadog and have had to turn off both due to cost. Comparing to our hand rolled solution, it's exponentially higher. I believe it's due to how the latency histograms uses individual metric tags to count RPCs in each bucket. This is both expensive and really not usable in datadog.

Are you going to collaborate with folks working on Kubernetes or other projects to find a way to automatically expose z pages, e.g. to the Kubernetes dashboard or the GKE console pages? That's one of the first things that ex Googlers miss from Borg. Under Borg you have to create a magic file with an HTML fragment, but there were APIs to help with that.

Developers and devops folks can plumb those manually, of course, using an Ingress object, but that's tedious and unneeded friction.

Yep! The Java library has some basic z-page support, but adding high quality z-pages will be a big part of the OpenCensus agent and OpenCensus service that we're developing now. These can be used within Kubernetes environments or on plain VMs.

High quality z-pages are great, but if there's no easy and secure way for a developer to get to them, they become useless. Rather than their contents or generation, my question is about automatic discovery and secure, restricted proxying of the pages. I don't want to tell my colleagues to run ad-hoc kubectl commands for each service or pod they want to inspect.

Ah, got it. We've discussed exactly this (within Google) with some of the Kubernetes maintainers who have similar wishes to yourself. The most immediate z-pages related work is focused on making them better in the more general case, as they're extremely barebones at the moment and are specific to each language's library. After that we'll explore the native Kubernetes integrations that you're asking for.

It'll come, but it's going to take some time.

Really cool project! Thanks for sharing.

Maybe a dumb question (and I certainly haven't read enough): Any plans or projects that decouple language support from exporter/backend support? Bummer to see a bit of fragmentation there, of course can understand architectural reasons for it.

I think that will eventually be solved by https://github.com/census-instrumentation/opencensus-service

Exactly.

There is! We’ll have an agent/sidecar available soon that means any exporter can be written in Go and only Go.

Man I would really love it if AWS got on board with this (not holding my breath though). After container orchestration, monitoring and tracing is the next major area that I would love to see a standardized API/toolset supported by all Cloud providers.

AWS also needs to revamp Cloudwatch. The UI/UX and agent integrations are severely lacking. Everything feels like it was built by a different team and like updates stopped as soon as they had something functional.

Google's vision with Stackdriver as a single pane of glass to handle logging, errors, metrics and tracing is awesome and as I write this I am seriously considering testing it out even though all my clients run on AWS. Google is clearly positioning Stackdriver for that use case anyway.

Another OpenCensus maintainer here, happy to answer questions along with Nik

How does this work with Istio? I believe Istio also collects traces and sends them to Jaeger/Zipkin.

How does OpenCensus integrate with existing tracing solutions like Jaeger or OpenZipkin? If I instrument my app with OpenCensus tracing today, which tools/services can I export this data easily?

You can send traces to Jaeger, Zipkin, Stackdriver, Azure App Insights, X-Ray, Honeycomb, and a few others. Supported metrics backends include Prometheus, Datadog, SignalFx, Stackdriver, Azure, etc.

How does the tracing API differ from the open tracing APIs? Pros/cons as compared to that standardization effort?

The APIs themselves are pretty similar. The biggest difference is in the scope/goals of the projects.

OT defines an API that various tracers can implement while OC ships a single implementation that tracing vendors can export data from.

OT is focused on traces, OC aims to support move observability signals. Traces and stats are available today. Logs and error tracking are on the roadmap.

There’s a few other differences, but those are the main ones.

"Open tracing" is a bit too opinionated by be universally useful. For example opentracing makes sampling decisions at the end of the span, not the beginning. Because of this you always pay the cost of span annotations even though the vast majority of spans are discarded. I think it's interesting that opentracing descends from the author of Dapper, and opencensus is from Google itself, and yet they've diverged on this and other details. Google of course just uses Dapper.

OpenTracing doesn’t actually specify when a span is sampled. IIRC Lightstep sends all spans to a service that makes sampling decisions based on moving latency percentiles, while Jaeger works as you described, sampling in the client. Its implementation specific.

We also use census for a variety of tasks, mainly for metrics but also for propagating small tags from task to task. The tags [1] can be essentially thought of as request-local scoped state for the entire tree of requests.

The presence of these tags can be used to control behavior if you're clever about it.

[1] https://opencensus.io/tag/

Reply on news.ycombinator.com