Observability for LLM apps with structlog and DuckDB | Hacker News Reader

Observability for LLM apps with structlog and DuckDB

ploomber.io

24 pointsedublancas10 comments

Still reading the post, but for anyone who is more interested in Structlog than this particular tutorial, here's their docs: https://www.structlog.org/en/stable/

Looks preeeetty cool. I rolled my own python logging system when I was using Textual and RichText, but the methods all revolved around style and flagging content. It never occurred to me structure logs in a way other than the (obviously, super noisy) call stack...

EDIT: Philosophy/"Why Structlog?" doc: https://www.structlog.org/en/stable/why.html

Structured logging is a thing; there are such libraries for other languages too.

This seems limited compared to the observability frameworks that integrate with the popular LLM frameworks like LangChain and LlamaIndex. When you use those, you want automatic traces for the nested calls hidden down the function call chain.

This really seems more like a Flask API tracing example with as LLM call as an endpoint. Dare I call it a shallow developer marketing piece?

re: LLM observability more generally, they mostly seem to be building their own full-stack suite, whereas I'd like them to ship to my existing Grafana LGTM stack. I'm keen to check out https://docs.openlit.io/latest/introduction and migrate away from LangFuse, but there is the feedback on model response -> training dataset that is really nice

If you are looking for automatic instrumentation that generates open telemetry standard traces - check out Langtrace, https://github.com/Scale3-Labs/langtrace. Got support for all popular LLMs, frameworks and vectorDB with high cardinality.

This still has the same issue that the docs say I need 2 more databases, not making use of my existing grafana LGTM stack

https://github.com/Scale3-Labs/langtrace?tab=readme-ov-file#...

Another blocker for me is that it appears to need an API key for trying it out locally or self-hosting

---

I'm comparing your project to OpenLit on this front

- https://docs.openlit.io/latest/installation#kubernetes

- https://docs.openlit.io/latest/connections/grafanacloud

1. You do not need 2 databases if you are only looking to export the traces to your grafana stack. If you are using prometheus as the TSDB, you can directly send the traces there without needing any of the databases to be setup locally.

2. Also, you do not need the API key. You only need to install the python or typescript sdk and use the custom exporter option - https://github.com/Scale3-Labs/langtrace-python-sdk?tab=read... . If you already have a OTEL exporter running, you don't need even that. Just initialize the sdk after installing and it will do its thing.

I am one of the core maintainers of the project. Do let me know if you have any questions.

If you are looking for ways to monitor LLMs which tie in with the rest of your observability stack, check out these docs [1] on monitoring LLMs with SigNoz. The tutorial with OpenLLMetry uses Otel format - [1] https://signoz.io/docs/community/llm-monitoring/

I'm using Garfana LGTM, not SigNoz.

HN generally appreciates if you account for this and only promote your project when it's relevant to the discussion.

The stronger need for LLM apps is for persistent response caching and reuse. Once this is available at scale with well-defined cache expiration policies, printing logs selectively is easy. The solution ought to also support individual key invalidation as needed.

I think it really depends on your need, if you're working on your local and trying to get something to work, caching, scale etc, might be an overkill.

Reply on news.ycombinator.com