nirga

Hey Marc :wave:

Would love to see you integrate and adopt this as soon as it makes sense to you. OpenTelemetry is a great and mature piece of technology and we should all be aligning around it now, while it’s still easy to do so.

The open source is free for all ofc. Our platform provides capabilities for monitoring and detecting hallucinations hence cost more.

Same ticket that gets you to install something like Sentry - you wanna see what's happening in production and get alerted when things go wrong

Thanks for the issues - I'll fix it! :sweat_smile:

Reg. Grafana and others - it's simple, just set the env vars - https://www.traceloop.com/docs/openllmetry/integrations/intr...

Super easy - you can just use the standalone instrumentations directly - https://www.traceloop.com/docs/openllmetry/tracing/without-s...

Yes, Traceloop is kind of a Sentry for LLMs

That's interesting! tbh you're not the first engineer I heard saying that!

You mean that enterprises would prefer on-prem, right? But if I'm an engineer working on a large enterprise, what's the difference between a SaaS providing an on-prem option, and an OSS? It's not like I can just git clone a repo and run it on my large enterprise cloud environment.

It’s hard to build a community around an open source if you’re not on GitHub imo. And if you’re not building an open source, than iiuc GitHub can’t use your code for copilot training so the main reason for not choosing GitHub according to the post isn’t relevant.

It amazes me that in 2023 this is not a solved problem by design of the language. Why go doesn’t adapt the “optional” notion of other languages so that if you have a variable you either know it is not null or know that you must check for nullness. The technology exists

It should work, but LangChain has many quirks so it can depend on which syntax you're using. Ping us on slack and we'll assist -

https://join.slack.com/t/traceloopcommunity/shared_invite/zt...

Yes, since we're using vanilla OpenTelemetry, you can set your exporter to whatever you want, including OpenTelemetry Protocol File Exporter. But I'd still use some sort of a dashboard, like Jaeger or one of the open source observability platforms like SigNoz or HyperDX that you can run locally.

Yes, ofc. The LLM instrumentations are just like all other instrumentations.

Sure, would love to! I'll ping you.

Note that these products aren't the same, even though they all fall under the category of observability - similarly to how you'd use Datadog AND Sentry, although both can be called "observability platforms".

I do think vendor locking is a key differentiator, which some of the reasons why OpenTelemetry succeeded in the first place. I know that my previous company switched to OpenTelemetry for exactly this reason. You get the flexibility of using any platform you'd want (since we're compatible with OpenTelemetry), so it's not vendor-locking you to a specific platform with specific capabilities. Why use any of the ones you mention - maybe Datadog is enough if your use case is simple?

But there are more advantages - you get much more than just observability to the LLM itself - you can see calls to vector DBs, network calls, DB queries, etc. - this can be extremely useful IMO for RAG and autonomous agents for example

Thanks! Related to what you're saying, I was actually expecting some reactions from devs who'd ask "why is it a separate repo and not part of opentelemetry from day 1?".

And for that my answer would be that I think having a separate repo would allow this to evolve in a more natural way, and faster (whereas OpenTelemetry, given it's massive adoption already, evolves much slower, with committees etc.).

Then, at some point when this is stabilized and useful - we can merge.

Kind of like Tesla's NACS vs. CCS

Wow, where do I start? The APIs are not that similar, but we're trying to use the same set of semantic conventions for everyone so for example you'll always get the model version, or the temperature in the same attribute. Which kinda means it's identical cross-vendor, at least on the o11y side.

Here are all the semantic conventions we've defined so far - https://github.com/traceloop/openllmetry/tree/main/packages/...

Yes, only traces for now. We do want to send out metrics for prompt length, token usage, etc. like you mentioned. Hopefully will be available soon (and we welcome contributions :) )

2 differences:

1. You don't have instrumentations for libraries like OpenAI, LangChain, etc. so you need to manually open spans

2. As you said, there are no semantic conventions for logging things like prompts and chains.

What we did is just defined the new set of semantic conventions, and built the instrumentations. But we're using vanilla OpenTelemetry so it's fully compatible with standard OpenTelemetry.

but it's open! :)

Yes, it's exactly the same setup. I think OpenTelemetry gives a whole world of possibilities you can build on top which are much more than just observability.

We're creating the backend equivalent of what cypress is doing. So in cypress you're generating FE tests where you click buttons and assert on FE components. Our framework can do that for checking BE flows. Let's say a user registered on your website, you can then create a test that make sure that there's a new entry for her in all your DBs, that you send her a welcome e-mail, and that you reported a BI event with all the information (like referrer etc.).

Yes that’s our long term goal. I don’t like having to adapt 4 different tools to do things which are quite similar.

I love growthbook! We’re building a different kind of feature flag framework - one that is for b2b and connects to your billing and pricing systems (because which feature is enabled is connected to what the user paid for).

Yes, we're actually working on supporting metering and billing these days. I think the golden path for a SaaS startup is to have as few products as possible.

We're currently working on metering and billing integration so I'm not sure if such integration makes sense.

I get it - that's why it's open-source. You can just run it on-prem.

What you suggested gets complex once you want to start making changes to your subscription plans. Let's say you want to experiment with different configurations. You'd then need to remember which customers were signed up with the "old" package configuration and each of the arms of your experiment, for example.

Thanks! So we want to tie it to pricing and tiers - the value of the feature should depend on which tier the customer's in.

Care to explain more?

Yeah I know :) I meant that there are some things specific to SaaS that we're trying to solve - like connecting it to pricing and package tiers. So if we have a feature flag called "SAML enabled" its value is based on what pricing tier that specific customer uses.