Hacker News Reader

Comment by bilalq | Hacker News Reader

Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads.

Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to.

In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and then have a log stream processor that then writes to your gateway collector. Other than that, it seems unavoidable, no? Sure, in something like Fargate you could go app to sink. But even that has its own tradeoffs.

View on HN

(I’m not the person you replied to, but have experience here.)

I follow the [gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/gateway/). Everything sends telemetry to our gateway, which exports to ClickHouse (formerly Datadog).

We use Node.js, so all we need to do is run a script initializing Otel before running the app. We set this up following the docs a few years ago, and haven’t had to change it much since then.

A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars).

The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink.

It's just outbound HTTP or GRPC, and it doesn't have to go over public Internet.

> In a runtime like Lambda, how would you avoid the need to run an edge collector?

Here's my setup (in Go, very simplified):

> // Instantiate a new slog logger > logger := otelslog.NewLogger("root", otelslog.WithLoggerProvider(otelLogger)) > // Use the logger as needed

My code uses proper Go loggers exclusively. I also redirected the stdout and stderr to a goroutine (via the usual close(2)+open() trick) to serve as a catch-all sink for anything that slips the net.

In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?

This is what I have done with CLI apps the directly send to the OTEL vendor. It works great.

Use the lambda layer [0] it sends the telemetry after the response is sent, so it doesn’t block.

[0] https://github.com/open-telemetry/opentelemetry-lambda

That lambda layer comes with an incredibly heavy performance penalty.

It doesn't block, but it does consume compute/memory resources and takes forever to startup[0][1]. To be fair, Rotel is promising in this regard[2].

[0]: https://github.com/open-telemetry/opentelemetry-lambda/issue...

[1]: https://github.com/aws-observability/aws-otel-lambda/issues/...

[2]: https://github.com/rotel-dev/rotel

I don't use Lambda anymore, but yes. I submitted traces to AWS XRay in a background goroutine with a small timeout.

If you're sending data purely to X-Ray, there's already a daemon running on lambda that you can forward to with low overhead if you don't use OTel. You also get near zero-cost logging and metric to Cloudwatch and EMF. But if you want bring destinations in the mix or do anything other than Cloudwatch , you have to pay the OTel tax. And even if you were content with a pure AWS setup, OTel is still being pushed on you now.

The X-Ray daemon and SDKs are all deprecated now in favor of OTel. Things like enchrichment of resource level traces for things like the DynamoDB client in v3 of the AWS JS SDK don't work with the X-Ray SDK. And they never will now. You're now recommended to use the AWS Distro for OpenTelemetry setup and OTel SDKs. The performance overhead of this is heavy, with big cold-start penalties.

Compare this with how the Datadog layer does adaptive flushing and performs relatively much better. Rotel is also promising in this space. But right now, OTel feels immature and things are being deprecated without the replacement being fully baked.

Why do you even _need_ these "layers"? It's a simple RPC protocol that submits data tagged with Span and Trace IDs.

That's really all there is to it. You can just submit it directly, without involving any layers.

Because blocking on OTel data to flush before sending a response back is often unacceptable. The layers run a standalone process using the Lambda extension API so they can keep running after your function has responded to a request.

Reply on news.ycombinator.com