Hacker News Reader

Hacker News

Comment by bilalq | Hacker News Reader

Parent Full thread

bilalq

In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?

View on HN

Use the lambda layer [0] it sends the telemetry after the response is sent, so it doesn’t block.

[0] https://github.com/open-telemetry/opentelemetry-lambda

That lambda layer comes with an incredibly heavy performance penalty.

It doesn't block, but it does consume compute/memory resources and takes forever to startup[0][1]. To be fair, Rotel is promising in this regard[2].

[0]: https://github.com/open-telemetry/opentelemetry-lambda/issue...

[1]: https://github.com/aws-observability/aws-otel-lambda/issues/...

[2]: https://github.com/rotel-dev/rotel

I don't use Lambda anymore, but yes. I submitted traces to AWS XRay in a background goroutine with a small timeout.

If you're sending data purely to X-Ray, there's already a daemon running on lambda that you can forward to with low overhead if you don't use OTel. You also get near zero-cost logging and metric to Cloudwatch and EMF. But if you want bring destinations in the mix or do anything other than Cloudwatch , you have to pay the OTel tax. And even if you were content with a pure AWS setup, OTel is still being pushed on you now.

The X-Ray daemon and SDKs are all deprecated now in favor of OTel. Things like enchrichment of resource level traces for things like the DynamoDB client in v3 of the AWS JS SDK don't work with the X-Ray SDK. And they never will now. You're now recommended to use the AWS Distro for OpenTelemetry setup and OTel SDKs. The performance overhead of this is heavy, with big cold-start penalties.

Compare this with how the Datadog layer does adaptive flushing and performs relatively much better. Rotel is also promising in this space. But right now, OTel feels immature and things are being deprecated without the replacement being fully baked.

Why do you even _need_ these "layers"? It's a simple RPC protocol that submits data tagged with Span and Trace IDs.

That's really all there is to it. You can just submit it directly, without involving any layers.

Because blocking on OTel data to flush before sending a response back is often unacceptable. The layers run a standalone process using the Lambda extension API so they can keep running after your function has responded to a request.

This is what I have done with CLI apps the directly send to the OTEL vendor. It works great.

Reply on news.ycombinator.com