OpenTelemetry
opentelemetry.io
opentelemetry.io
For example, if you visit the OpenCensus site, it tells you on every page that OpenTelemetry is the way to go now. Yet OpenTelemetry's Java metrics implementation is still listed as alpha, and functionality as basic as tags didn't work when I last tried it.
Or, I've been working with Datadog lately, and I want to add dynamic span metadata. My options appear to be either to wire up the (deprecated) OpenTracing client, or set up a full collector/agent suite of OpenTelemetry processes (whose default tutorial configurations appear to be invalid in places).
worth I guess, but bumpier than I'd like.
It's been a lot of change and I can't wait for things to settle down and for companies to stop trying to get their piece. I feel like everyone (Datadog, NewRelic, Google, etc.) saw Lightstep as a competitor and wanted to get involved with a competing spec, which is how we ended up in this consensus mess.
Still, it's good progress and it's so much better than raw logs. Would recommend.
Logging is still experimental in the spec. Metrics API is feature freeze and the protocol is stable, so it's more on language SDKs to stabilize their implementations. This is a focus for several of them right now.
Only complaint is the churn over the past year, especially in the metrics world. We use OTel Python and have had to pin our libraries to an older version so we can continue to record metrics stably until the python SDK catches up to the latest metrics specification.
Historicaly APM (application performance monitoring) also owned collection, vendor locked-in and didn’t evolve price per value.
Open-source collection standard allows users to choose or switch to the best vendor.
Personally, I recommend Sumo Logic (disclaimer: I work there). We have rich full-text query capabilities on spans with parsing and aggreagations on query time…
1. Free plan includes Tracing.
2. There is monthly self-serve subscription available.
Lately I am leaning into managed more and more - I want to spend my time building great dashboards, or analyzing my data, or writing some code for something else entirely. If you run your own stack you are always keeping up not doing the really useful stuff. Getting a working baseline is hard enough, let alone shiny extras. You’ll watch the talks at whatever conference, you’ll read the single line “quick start docs”, it will sound amazing. Then you’ll find yourself deep in months later with a giant data transfer or storage bill and hours of painstaking tuning ahead of you.
If you’re really large, then you can go back to running it yourself.
It's just too hard of a sell for management. I don't even want to calculate how many times the cost of S3 storage that is. And then add on top of it that you can't gauge how much data you'll generate until you've actually started implementing it.
I don't think distributed tracing will become mainstream until there's something like ELK for it that lets you start out in-house. Then people start to see the value, and you have a solid case when maintaining it becomes a full-time job.
I haven't seen anything that seems production-ready enough to use at work, but also doesn't involve running an entire Cassandra cluster or something. Is there an implementation that can run using Postgres? I have no doubt that it wouldn't scale well, but I need a foot in the door to prove the value before I can justify paying $93/GB to my director.
As for the crashing scenario, this seems like an application-level concern. Ideally it is not waiting until the crash to send traces. Depending on the environment, the application could handle the crash and deliver the telemetry leading up to it before shutting down.
[1]: https://github.com/open-telemetry/opentelemetry-specificatio...
In terms of crashing and taking a long time to complete, these are problems that are very difficult to solve on the backend as well. Simply having the start event without the end event is not enough info to say for sure that there will _never_ be an end event. For low data volumes and simple use-cases this may not seem like a big deal, but it gets complex extremely quickly.
However, this design "flaw" is well known and seems assumed. I'm not able to find relevant GitHub issues right now, but I remember this topic being discussed on OpenTracing or OpenTelemetry bug tracker and the outcome was something like "Spans might not be the best data model, but people are now used to it and we have to ship the spec within a reasonable time, so let's stick to it".
Edit: https://github.com/open-telemetry/opentelemetry-specificatio... might be relevant to your concerns.
I wonder if it's worth it to replace both metrics and logging just to take advantage of OT's tracing capability.
How would you say does opentelemetry compare to Qualtrics?
Is Qualtrics a tool where users need to actively fill out surveys and opentelemetry collects passively user interactions?
There is also other names floating around in this space like DataDog or OpsTrace, where I am not clear how to place them.
When customers go through this workflow step, they use this option/button 90% of the time and the other options are used rarely?
I didn't think to much on this but I played with it once in a hackathon and made a tool that showed how one customer site performed versus another customer site by showing performance comparisons on workflow spans and augmenting this information with metrics of most used tools to get an indication if the other site was maybe faster because it used different parts of the product to perform certain actions faster.
I see a telescope. Is it space related? Telemetry of satellites?
Just my two cents as someone not familiar with the project, and has no idea how to use it or how it would solve a problem I have.
> OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.
[0] https://github.com/open-telemetry/opentelemetry-specificatio...
And tracing is GA in most SDKs as well, so you should feel ready to adopt it. Several vendors and OSS tools support it (or you can export your data to another format with the OTel Collector).
Please stop helping spy on users! Be the change you want to see in the world!
It's a set of tools and a specification for instrumenting applications and collecting logs/metrics/traces from them. It's used to ask questions like "why is this page slow to load?" and get answers like "because it talks to the billing microservice which has a really slow SQL query".
This isn't the same sort of telemetry that desktop software vendors collect.
"the browser"
That means they are running code on my machine and exfiltrating data out.
If this is implemented in a totally transparent, user controlled, consensual way, and the collected data made ephemeral and subject to user request for deletion and viewing on demand while being stored, then there's nothing bad going on.
Anything less is unacceptable - a line is being crossed. Right now a majority of the encounters users have with telemetry is totally opaque and behind the scenes, nonconsensual except for the barest shreds of a EULA or button click fig leaf.
I'm curious why you automatically assume data is being collected about the user? OpenTelemetry is about Observability, not user tracking.
In some ways, such information is just as important as biometrics and passwords and pii.
Deanonymizing becomes possible when metadata is cross-referenced. Metadata should be subject to as strict protection and consent rules as documents, health info, or any other "obvious" private data.
If it's running on your server and not logging third party activity or metadata, the information belongs to you. Else, you need the third party's consent, etc.
"It's just for benign development purposes" doesn't cut it anymore regardless of intentions. We need a cultural and legal area change with regards to privacy and primacy of private data protections.
I understand it seems extreme, but it's really not.
Any data generated by or on a user's device is private data. Recording that data is surveillance. No matter how harmless it might seem, data collection should be consensual, transparent, and ephemeral. You should specify, explain, and obtain consent for any and every variable. Anything you do with the data should be reportable to the user. Anything less creates opportunities for abuse.
Logging ip address connections is a great example. It's simple and trivial but how many RIAA piracy lawsuits do we have to see inflicted on innocents before we decide keeping those logs might not be a great idea?
The misuse of metadata by law enforcement through plausible narrative crafting is ubiquitous. It's not about the developer's intentions, it's about collection of surveillance records that can be seized or stolen and weaponized.
OpenTelemetry is a vendor-neutral framework and standard for generating performance data (spans, metrics, logs) about systems, particularly distributed ones where it's impossible to just load up a debugger to figure out why something is failing or slow. It's not a framework for collecting user data. User data and performance data are completely separate things.
Your concerns about privacy are important, but not really relevant to the topic.
What’s with the assumption that the service provider cannot observe how their system performs?
In a similar way, your software is borrowing time and computing resources from your users. You need to use them sparingly and respect that you do not own their hardware, but are a guest they have invited in. It's not your computer to do whatever you want with.
The OpenTelemetry project is focused on Observability use cases and not user analytics/tracking/etc. The fact that of all the libraries, a JS browser library exists doesn't automatically mean the parent comment is "wrong".
I suggest anyone making snap judgements about OpenTelemetry spend a few minutes reading more about the project. This general pattern of comments surfaces every time this project is posted here.
OT is for distributed tracing across services. It's a glorified profiler (not to downplay how awesome OT is). Has nothing to do with spying on end user behavior any more than URLs, JavaScript or HTTP headers has to do with spying.
Could you use it to spy? Yes, you can use anything to spy, if you choose to.