pranay01

I think the biggest value I see with OpenTelemetry is the ability to instrument your code and telemetry pipeline once (using otel collector) and then choose a backend and visualisation framework which meets your requirement.

For example at SigNoz [0], we support OpenTelemetry format natively from Day 1 and use ClickHouse as the datastore layer which makes it very performant for aggregation and analytics queries.

There are alternative approaches like what Loki and Tempo does with blob storage based framework.

If your data is instrumented with Otel, you can easily switch between open source projects like SigNoz or Loki/Tempo/Grafana which IMO is very powerful.

We have seen users switch within a matter of hours from another backend to SigNoz, weh they are instrumented with Otel. This makes testing and evaluating new products super efficient.

Otherwise just the effort required to switch instrumentation to another vendor would have been enough to not ever think about evaluating another product

(Full Disclosure : I am one of the maintainers at SigNoz)

[1] https://github.com/signoz/signoz

Thanks for mentioning SigNoz. Totally agree that OpenTelemetry and open source should be the way forward here.

If anyone wants to check out SigNoz repo - https://github.com/SigNoz/signoz

PS: I am one of the maintainers at SigNoz

If you are looking for ways to monitor LLMs which tie in with the rest of your observability stack, check out these docs [1] on monitoring LLMs with SigNoz. The tutorial with OpenLLMetry uses Otel format - [1] https://signoz.io/docs/community/llm-monitoring/

You should also check out SigNoz [1], we are an open-core alternative to DataDog - based natively on OpenTelemetry. We also have a cloud product if you don't want to host yourself

[1] https://signoz.io

Thanks for mentioning SigNoz. Here's our github repo in case anyone wants to check it out - https://github.com/signoz/signoz

.. Of Grafana labs , by Grafana labs, for Grafana labs :)

You might want to have a look at SigNoz [1] as well. We have also published some perf benchmark wrt Elastic & Loki [2] and have some cool features like logs pipeline for manipulating logs before ingestion

[1] https://github.com/signoz/signoz [2] https://signoz.io/blog/logs-performance-benchmark/

Just finished reading Exhalation by Ted Chiang, and can't recommend it enough. It's a collection of short stories - so easy entry point for beginners as well.

You can configure tools like Elastic or SigNoz[1] for this.

https://github.com/signoz/signoz

What's the issue with protobufs?

Agree, there being an open standard for instrumentation is a big win. Lots of work still needs to be done on showing more examples and making it more accessible to users & implementors.

One other key area is resources which can help get engineers/implementors to get organizational buy-in

SigNoz maintainer here.

We also have traces, metrics and logs in a single application which makes correlation across them much easier. From what I can understand from Quickwit website, they use Grafana and Jaeger for UI.

Here's our github repo if you want to check it out. https://github.com/signoz/signoz

hey - SigNoz maintainer here.

> it's comparable to Datadog feature wise but nowhere close in performance and scalability

Would love to understand how you tested SigNoz and what were the issues you found in performance and scalability?

Hey - SigNoz maintainer here.

we primarily focus on Application Performance monitoring (APM), Distributed Tracing, Logs management, Infra monitoring & Exceptions monitoring.

We have some features for Client side monitoring, but it's not as advanced as of now (though it's in our roadmap)

You can do basic client side monitoring if your application is in JS though - https://signoz.io/docs/tutorial/instrumenting-angular-fronte...

Have not tried myself but heard from multiple folks that self hosting sentry is not easy

Awesome! Do reach out to us in our slack community[1] if you have any questions or need any help on setting things up

[1] https://signoz.io/slack

Yeah, the ability to send to multiple sources is quite powerful and most of this comes from the configurability of Otel Collector [1].

If you are looking for a open source backend for OpenTelemetry, then you can explore SigNoz[2] (I am one of the founders) We have a quite a decent product for APM/tracing leveraging opentelemerty native data format and semantic convention.

[1]https://opentelemetry.io/docs/collector/ [2]https://github.com/SigNoz/signoz

I think that OpenTelemetry will solve this problem of vendor lock in. I am a founder building in this space[1] and we see many of our users switching to opentelemetry as that provides an easy way to switch if needed in future.

At SigNoz, we have metrics, traces and logs in a single application which helps you correlate across signals much more easily - and being natively based on opentelemetry makes this correlation much easier as it leverages the standard data format.

Though this might take sometime, as many teams have proprietary SDK in their code, which is not easy to rip out. Opentelemetry auto-instrumentation[2] makes it much easier, and I think that's the path people will follow to get started

[1]https://github.com/SigNoz/signoz [2]https://opentelemetry.io/docs/instrumentation/java/automatic...

You should also check out SigNoz [1]. It's an open source observability platform with metrics, traces and logs in a single application & based natively on opentelemetry.

One of the reasons many people use SigNoz is to avoid the vendor lock-in which comes with adding proprietary SDKs of closed source products like DataDog and New Relic in their code.

If anyone is starting their observability journey today, I think OpenTelemetry is a very good place to start. You can instrument with Otel SDKs and chose a visualization layer/backend which suits your needs best

(Disclaimer: I am maintainer at SigNoz)

[1]https://github.com/signoz/signoz

Have you seen OpenTelemetry logs?https://opentelemetry.io/docs/concepts/signals/logs/

I see, can you explain. Aren't transistor sizes still reducing which is what Moore's law predicted (being able to fit more transistors in a wafer)?

is that actually possible?

So, is 3nm not thinner/smaller in size compared to 5nm?

Do you mean sampling at application level before sending traces to otel collector/Kafka?

Querying in Tempo/Loki does seem to not scale particularly well, and Loki has known issues with high cardinality data, so...

Have you done/seen any benchmarks between Redpanda/NATS and Kafka for this use case?

Some folks in SigNoz community have also suggested NATS for this, but I have not deep dived into benchmarks/features yet

this is an interesting tutorial on how to run OpenTelemetry at scale. When ingestion volume becomes super high, a queue like Kafka seems to be important to make otel collector handle spikes, etc.

Interesting to see comparison between pricing for different observability tools

hey, thanks for sharing it here.

PS: I am one of the founders at SigNoz

Thanks, SaaS does have more features like SSO/SAML login, more advanced trace analytics features, and of course it's hosted by SigNoz team.