Pydantic Logfire
pydantic.dev
pydantic.dev
The llm integration seems promising but if you care about LLM observability you probably also care about evals, guardrails and a million other things that are very specific to LLM's. Is it possible to build all this under one platform?
I do hope I'm wrong for the sake of pydantic-core.
I think Pydantic is great software and so I am inclined to see if this too will be great software.
My current company, a much small startup, primarily uses Datadog and we are starting to better integrage Honeycomb. We mostly abandoned Google Cloud Monitoring because the UI/UX are not that great. Honeycomb is a paradigm, so took some time for us/me to understand. It's growing on us.
Despite the ability to completely blow up our bill (which is now better controlled), Datadog is a good product that lets us quickly answer questions when things go wrong. It's not perfect, but it's the best we have right now. The UI is intuitive, and facilitates more discovery (esp. for metrics and their attributes).
(esp. for metrics and their attributes).
We also support logging as an integrated concept into tracing (you can emit events without a duration that are just like a log but carry context of where they were emitted from relative to a trace).
Gotta echo the sentiment that Logfire doesn't seem to be too closely related to Pydantic... Also, afaict it looks like the frontend is not open source, unless I'm missing something [2]. So, not a tool that one could self-host?
--
Operating AI apps reveals a big challenge, in that debugging probabilistic code paths requires more than the usual introspective abilities, and in an environment where function calls can have very real monetary impact we have to be able to see what’s happening in the runtime. See LangChain’s hosted solution (can’t recall the name) that allows an operator to see prompts and responses “on the wire”. (It just occurred to me that Langchain and Pydantic have a lot in common here, in approach.)
Having a coupling between Pydantic - which is *just about* the data layer itself - and an observability tool seems very interesting to me, and having this come from the folks who built it does not seem unreasonable. WRT open source and monetization, I would be lying if I said I wasn’t a little worried - given the recent few months - but I am choosing to see this in a positive light, given this team’s “believability weight” (to overuse Dalio) and history of delivering solid and really useful tooling.
I understand why this might be your reaction, but let me just share my thoughts:
Can we build all the LLM things people want? Yes I think so, early feedback is that Logfire is already much more comprehensive than LLM specific solutions.
How is our solution any different? AFAIK:
* no one else offers opinionated wrapper for OTel to make it nicer to use, but with all the advantages
* no one else does auto-tracing (basically profiling) using import hooks the way we do
* no one else has dedicated instrumentation for OpenAI or Pydantic
* no one else provides metadata about python objects the way we do so we can render a semi-faithful repr of complex objects (Pydantic models, dataclasses, dataframes etc.) in the web UI
* no one else (except Sentry, who are doing something different) makes it really easy to start using it
* no one else lets you query observability data with SQL
In my mind, the obvious alternative would be to do the thing most OSS companies do, and build "Pydantic Cloud", then start adding features to that instead of the open source package. I didn't want to do that, and I don't think our users would like that either.
In the end I decided to build Logfire for two reasons:
1. I wanted it, and have wanted it for years
2. I think building a commercial product like this, then using OSS to spread the word and drive adoption is a really exciting way to incentivize us to make the open source as good as possible — good as-in permissive and good as-in well maintained. And it means that our commercial success is not in tension with the adoption of our open source, which has been a recurring issue for companies trying to capitalize on their open source brand.
Sadly I've moved on from Python world.
Funny anecdote, using Pydantic everywhere to improve maintainability made me realize I'm fighting an uphill battle with Python and I should move to a statically typed language, so I switched to C#.
Thanks for your work.
I've actually really wondered why one wants static typing, especially with a language as expressive as Python, and where you can be so efficient with it?
At a time, I was mesmerized with Zope (esp 3), but learned that nobody really learns to use it, but instead learns to work around it — for the uninitiated, Zope provided "interfaces", you'd have classes implementing those interfaces, and you could happily mix and match with "configuration" that lived in ZCML (an XML schema) files, way back in early 2000s.
Static typing really kills off some of the biggest benefits of using Python (like "duck typing" to quickly emulate an identical API without having to construct a hierarchy of types before you can do that).
How many bugs have you really hit in your Python code because of lack of static types? For ~20 years of doing Python, I honestly believe that it could have helped me at most once or twice. Generally, doing sufficient level of testing has covered most potential misuse of code, and you need to have tests anyway.
The reason I'd want to move away from Python is mostly pure performance (loops should not be this slow) and library ecosystem (crappy code has risen to the top), but I find nothing is nearly as expressive, has a comparable standard library plus allows one to be so efficient.
I don't work with Python much anymore (happily moved on to Go :), but was a heavy user for ~15 years. And honestly, typing issues were pretty common, if not the most common issue across all codebases I've worked on.
The problem with dynamic typing is that, at the end of the day, you're still working with and thinking about types. You have to be implicitly aware of which type the function you're calling expects, which not only increases your mental burden, but makes refactoring much more unreliable. You have to rely on documentation or _very_ thorough tests, which most codebases don't have to the extent and quality they should.
With static typing, all of this goes away. Types become explicit (remember "explicit is better than implicit"? Yeah...), you get immediate feedback from your IDE when passing the wrong type, and the code simply doesn't compile.
Best of all, you don't learn about a TypeError exception from a Sentry alert after your users run into it. The amount of times this happens in the wild is shockingly high.
> For ~20 years of doing Python, I honestly believe that it could have helped me at most once or twice.
I honestly struggle to believe this, but good on you if true.
> Generally, doing sufficient level of testing has covered most potential misuse of code, and you need to have tests anyway.
But that's the thing: in order to catch basic typing issues, you would have to have dozens of tiny and mostly pointless tests for each function. Yes you need tests, but most tests, even at the unit level, shouldn't be concerned about types, but about the functionality. And you won't catch typing issues at the boundaries in integration, or higher level tests. Static typing simply helps you avoid all of this nuisance, and ultimately write more robust and maintainable code.
If I was ever to work on a Python codebase again, I wouldn't consider it without a runtime type checker like Mypy, or whatever state of the art tooling is these days. All the supposed freedom of dynamic and duck typing just isn't worth it.
Having seen some of the codebases in the wild as I moved projects, I can understand why someone would feel that way — and to be honest, I was referring to the code I wrote, not necessarily code that someone else wrote, where I've seen many type errors, but which were really either irrelevant (exceptions that should have been "nicer" errors, as in, this was bad data getting passed in between systems), or missing functional tests (see below).
Notably, once I moved to a Go codebase, I uncovered a weird bug that could have been caught by typing, but really wasn't due to induced type complexity by the developers to get things to work at all — mostly to demonstrate that complex type structure likely leads to bugs, rather than typing or lack of typing.
> But that's the thing: in order to catch basic typing issues, you would have to have dozens of tiny and mostly pointless tests for each function.
Yes, that's the thing: I am specifically not referring to the "typing" type of test like "this throws an error if an int is passed in instead of a str" or BaseFoo instead of BaseBar, but really, a test that confirms something fails when it needs to fail, and works when it needs to pass — iow, regular functional tests that you need either way.
> Static typing simply helps you avoid all of this nuisance, and ultimately write more robust and maintainable code.
I fully accept that may be true for some codebases, but I don't think it's true for everyone. My biggest gripe with Python is that idiomatic (or maybe "widely accepted way to write") Python leads to less robust and maintainable code, and non-stdlib packages are really, really, crappy. Django does like a gazillion of those anti-patterns, but even things like SQLAlchemy, Requests, Flask, FastAPI are problematic.
I had the (mis?)fortune to land my first real job in a wonderful TDD shop way back in mid-2000s, and after I survived 6 months without getting fired (and it was close, I heard later :)), it was an excellent learning opportunity.
Being explicit about types allows the IDE and platform to help you write & refactor code, allowing your brain to work on the actual problem instead.
This may sound surprising but modern C#/.NET is almost as expressive as Python. You don`t need unnecessary boilerplate or AbstractBaseFactorySingletonProxyDecoratorAdapterWrapper style types. LINQ alone is a game changer. I`m still writing code with almost same style as Python in C#, but now with the power of IDE behind me. I`m much more productive.
Interfaces are duck typing in an explicit form. Being explicit about it may sound more work, but you also need that work in Python if you want your solution to become maintainable, in the form of tests.
I`ve been using Python for ~10 years (with years of PHP/Ruby before) and written probably hundreds of thousands of lines of code. One thing I don`t like while programming is doing unnecessary work myself if the computer can do it. So I do a lot of meta programming and build abstractions when it becomes annoying to repeat myself. There had been a lot of moments where I return to a highly dynamic part of code that I`ve written months before to fix a bug and spend a lot of time trying to figure out how everything is connected. Same code in C# is both quicker to write (once you grok how reflection works and how everything is tied together), and infinitely more maintainable for less work.
Give C#/.NET a serious try. You may become surprised. I was.
That's very curious: do you have any quick examples that you could point me to?
I was generally intrigued by C# and .NET way back when it was introduced and especially when Mono started shipping the runtime for Linux, but I hated the boilerplate and couldn't move away from Python for that reason.
This is not to say that one should use string concatenation to construct queries, but that you want smart tools that do just enough. I.e. I like Storm expression language or sqlalchemy-core to express SQL as Python, but getting into deeper ORM level stuff generally harms what you can achieve once you need more complex stuff.
Maybe it's the Python disease of trying to be everything at once, which can end up as the worst of all worlds.
Ruby is a proudly untyped language, and that's all it is.
It is this easy to make a back-end application with ASP.NET Core: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/m...
Certainly seems like the Logfire initializer could coexist with code like in https://docs.honeycomb.io/send-data/logs/opentelemetry/sdk/p... - is this a reasonable assumption? We're looking into modernizing our Honeycomb integration from their older direct integration to their OTel SDK, and would be very curious if Logfire can exist alongside this. Do you do any patching of OTel internals that would make this impossible?
I don't think we currently expose an easy way for you to send data to two (e.g. honeycomb and logfire), it it should be entirely doable, and if people want that, we can make it easy.
[1] https://github.com/Scale3-Labs/langtrace [2] https://github.com/open-telemetry/semantic-conventions/blob/...
The most important thing is that you share with us WHY you want to self host. Is it compliance? What does that compliance mean (min/max data retention, right to be forgotten, data at rest needs to be geographically located or you have to own it, etc)? This will help us build the “right” kind of features in this area so that it works for you and your company.
In my case, I am hesitate to add another cloud service in the Sub-Processor list to meet GDPR.
Actually, if it is an Otel wrapper, I don't think you don't need to offer self-hosting the server at all. Making easy to integrate with other Otel ecosystems would be enough.
> It would be more of a licensing setup.
At least this is much better than FOSS first than switching to a non-FOSS license like what happened to hashicorp and redis.
I like the prefect.io approach to have the control plane on the external provider, with the data and workers being run on the customer infrastructure. It seems fair for both sides: - as the subcontractor, you keep trace of the real usage, without having to handle end user data which is a pain to manage, so you don't have to offer outrageous license pricing to compensate for being stolen (looking at you Grafana) . - as the company, you comply with gdpr, while alleviate the operating costs, and also supporting the companies providing the tooling you need.
Compare https://github.com/getsentry/self-hosted/blob/9.1.2/docker-c... with https://github.com/getsentry/self-hosted/blob/24.4.2/docker-... for what life used to be like for running Sentry on-prem. It was awesome
It would take a ton of work to dig up the actual memory and CPU requirements of each one, but rest assured they're not zero, so every one of those services eats ram and requires TLC when, not if, they shit themselves. So, more parts == more headaches with all other things being equal
Then, I deeply appreciate that there are a whole spectrum of reactions to the various licensing schemes in use nowadays, and a bunch of folks don't care. I care, though, because I have gotten immense value from open source projects, and have contributed changes back to quite a few. It has been my life experience that many of those "source available" licenses usually are very hostile toward making local real builds and if I can't build it to match how prod goes, then I can't test my fixes in my environment and then I can't contribute the PR with any faith
Live and die on your hill. We'll keep focusing on building our product - which requires us to be able to be able to pay developers for the enormous amount of time it takes them.
I ordinarily would have just ignored your troll comment, but this was so incredibly short sighted that you were obviously wanting some sparks so now you'll get them. Sentry didn't start life with a source available license, even though the threat model to the business was exactly the same at that time: the cloud for sure existed, I know because I ran Sentry self-hosted upon it. And your cited enormous amount of time and money to pay developers didn't spontaneously spring into being 6 months ago, either. So, the tone that was set was that Sentry was open source with all the rights and privileges that came with it. Until someone got butthurt and decided they needed not just one source available license but then their own source available license just to ensure lawyers never go hungry
> Live and die on your hill.
You, too. Enjoy your mansions and yachts from all the ontold riches that your new licensing scheme will surely bring you in exchange for lighting fire to any trust gained
So lets talk about the facts, because the paint a pretty clear narrative, rather tha one that people would prefer to believe.
1. I built most of Sentry (back then), and while we had a few contributions here and there, it was almost exclusively my time, or future employees times. So no community contribution concerns.
2. We relicensed because of a new threat, not one that existed 16 years ago when I started the project. That threat was GitLab, who was openly trying to commercialize Sentry. They never once contributed to the project, nor did they want to contribute back as part of that strategy. I know this to be true because I asked them to.
3. We built the FSL because the BUSL did not create a strong enough conviction to our values - of which we repeatedly have put words into action on. We wanted to cement those values, and make it easier for people who had our same concerns, but also wanted to create more open source, to be able to achieve that _without_ undue risk or legal fees.
There is a huge difference in the way Sentry operates, and the way some of these other organizations have chosen to relicense (or in some cases, legitimately rug pull).
So you can say what you will, but we've always been straight forward with our beliefs, and talk about these things publicly all the time. I'm not here to convince you of changing your beliefs, but I will never sit idly when people spread false information, especially about us.
We decided to self host sentry, which is an absolute beast to deploy, the open-source helm chart is nowhere near production level, and the underlying technologies are quite hard to maintain (Kafka, zookeeper, clickhouse...). We had to work on it constantly for two months to stabilize it, and now fear the moment we'll want to update. The dev teams love it, so it was worth the hassle!
I think there's a less-misleading way to use open-source reputation for business credibility, though: "so-and-so business, by the creators of so-and-so project". Knowing that respected and skilled folks are working on a business is great! It's a fine distinction, but I think it matters.
Other than that, see my answer to the other comment.
But to be fair, the branding here is a little weird. If you were living under a rock and hadn't heard of pydantic then the website reads like pydantic is the company and logfire is a product of said company. That's fine. But then you've also got a product called pydantic. Or is it now called pydantic pydantic? I realise it's kind of an extension to pydantic but the AI focus doesn't make it feel that way and so I think that's where the GP is coming from. Sorry to nitpick, I hope it does well.
Google was originally just the search engine, without any qualifier. Now it's Google Search. In the same way, pydantic can become "Pydantic Validate" or something.
We could have called the company a completely new name and everyone would have been confused, or just called us "pydantic".
I think Pydantic being the company name, and a standalone entity, and there being other products is fairly common. I think if Logfire is successful, it will end up just being known as "Logfire".
I don’t see how anybody would have been confused in that scenario? People start new companies with new names all the time.
What you are actually doing is what is confusing. It’s a really weird branding choice to name an observability platform startup after a validation library.
From the team behind Pydantic, Logfire is a new type of observability platform built on the same belief as our open source library — that the most powerful tools can be easy to use.
We also have explicit integrations with popular logging packages (see the bottom of https://docs.pydantic.dev/logfire/integrations/#opentelemetr...), so if you use the standard library `logging` module, `structlog`, or `loguru`, you should be able to set up your logs to ship to Logfire with minimal modifications to your existing code.
While we have not announced our pricing yet, I am confident that tens of millions of log lines a month will absolutely be economical to store.
(Also, I'll just add that we also support metrics, though our metrics support is not as well-integrated yet as tracing. But soon!)
Try not to surprise us on pricing, though.
Could you clarify if it's possible to self-host the service?
I want the entry level for "enterprise" to be much lower than some other companies in this space.
(That being said, I am a sucker for all new tooling in this genre and am excited to play around with it!)