Show HN: LogLayer – Unified logger that routes logs to various logging libraries | Hacker News Reader

Show HN: LogLayer – Unified logger that routes logs to various logging libraries

loglayer.dev

56 pointstheogravity26 comments

As a longtime TypeScript/Node.js developer, I've often faced challenges with logging—choosing, using, and maintaining the right logger for various projects. While most loggers offer the usual methods like "info", "warn", and "error", they vary significantly in how they handle structured metadata or Error objects. This can lead to ad-hoc solutions, like serializing errors or writing custom pipelines, just to get logs formatted correctly.

I built LogLayer to address these pain points by introducing a fluid, expressive API. With methods like "withMetadata" and "withError", LogLayer separates object injection from the log message itself, making your logging code both cleaner and more maintainable.

Logs are processed through a LogLayer Transport, which acts as an adapter for your preferred logging library. This design offers several key advantages:

- Multi-Transport Support: Send logs to multiple destinations (e.g., DataDog and New Relic) simultaneously. I've personally used this feature to ship logs directly to DataDog without relying on their APM package or sidecars.

- Easy Logger Swapping: If you’ve ever used Pino with Next.js, you might have encountered issues where it doesn’t work out of the box after a production build without webpack hacks. With LogLayer, you can swap in a better-suited library without touching your logging code.

I spent a good few months on and off and used my winter break to launch version 5 of LogLayer, and also created the documentation using Vitepress.

LogLayer has been battle-tested in production at Airtop (https://airtop.ai), where it’s been an integral part of our systems for years (we were running as Switchboard for almost four years and pivoted late last year).

(Disclaimer: I work at Airtop, but LogLayer is not sponsored / affiliated with them.)

Open articleView on HN

IMO until the abstract logging interface is built into the language, or standard library, we'll keep building different log libraries and adapters between them, and shipping libraries without logging.

It makes me actively disinterested in these libraries. I just want some built in interface with levels, hierarchy, and structured logging

This is one of the things Elixir got right on day 1, all the libraries in the ecosystem use the standard library and it makes dealing with logs so much easier compared to other languages.

This slows down innovation.

In Java it is common to use the MDC support of slf4j. You can add to metadata in thread local variables, so that you don't have to add it at every logging site.

Any precedent for language agnostic behavior (I genuinely don’t know)? Would be keen to have a common interface that languages implement into their standard libraries or keywords

OTEL perhaps

Java has the java.util.logging classes.

Java historically has had so many logging frameworks it needs logging framework adapters to deal with the problem that in any significant program you will end up with dependencies that log in incompatible ways :/

It probably has the most complex and messy logging story of any language - but at least the solutions are very mature at this point.

Not a good example, as the de-facto logging interface for Java was SLF4J.

Though I myself prefer Log4J2, as I'm already using Log4J2 as the concrete logging implementation.

Why do you prefer Log4j2 over the slf4j api and why over logback as the backend?

I prefer Log4J2 built-in JSON Template Layout to Logback's various JSON output solution.

The API of Log4J2 and SLF4J v2 are similar enough that it does not matter for basic usage, I don't see the benefit of SLF4J in a project that's already using Log4J2.

Swift has a de-facto logging library: swift-log (by Apple, with levels and structured logging; not sure about hierarchy as I don’t know what you mean by that).

Probably, a hierarchy of different loggers.

You're right that people will continue building their own libraries. I also wish this was all built-in so you wouldn't need something like loglayer.

This is how .net does this as well. You write logs via ILogger interface and then wire it up to a logging library of any flavor.

ILogger is not in the BCL. Its part of the Microsoft.Extensions family of libraries.

Though today much of the Microsoft.Extensions family is considered BCL by most users, especially in the way it is documented as "Core functionality" (the "Generic Host" model uses a lot of Microsoft.Extensions, up front in .NET documentation today) and also in the way that the majority of it (especially in this case of Microsoft.Extensions.Logging) is source controlled side-by-side with most of the BCL, including almost all of System.**: https://github.com/dotnet/runtime/tree/main/src/libraries

(There are some of Microsoft.Extensions that exist outside of dotnet/runtime in dotnet/extensions: https://github.com/dotnet/extensions/tree/main/src/Libraries)

Never really caught on with the idea of integrating external I/O in something as fundamental as logging. More often that not a pull vs push approach is suited for logging. There are dozens of high performant log collectors/aggregators who can collect, transform, and push logs to N number of stores. What’s the advantage of doing this right inside the app? (Besides maybe serverless functions?)

Sometimes you don't have the resources or need to spin up an external collector. I might be in a situation where I want to collect logs with DataDog, but I also am not big enough nor would I have the time to set up a sidecar or configure the agent itself. It's possible you don't have access to the host environment to install something like an agent either (as you mentioned, like serverless functions).

The advantage for the in-app case is that it's fast to on-board with and doesn't require significant / any devops involvement compared to the above scenario.

One downside though is depending on how those logs get shipped, it might not scale well if you're writing an extreme frequency of logs (it's possible to reach HTTP limits with DataDog and New Relic log publish APIs in a high volume/frequency situation), and that's when you would want to move to and external collector that can process logs in a much more asynchronous fashion.

I'm not sure if the question is specific to LogLayer or just talking about logger transports in general that does in-app push, but I see the cloud collector features of LogLayer as something that you can easily add to your logging stack when you start out small and need it, but transition to an external collector later.

(One might argue that some OTEL tools can also do this, but as stated in another response, I'm not familiar enough to know if they'd do a better job or not in terms of the overall logging DX that LogLayer provides; their primary job is to ship logs from my understanding.)

+1 on this. Kids, don't rely on log to remote inside the application.

You get a growing combination of N apps x M log servers to support and link into the application. Even if you only use one of them yourself. Ops should be able to change to a new logging backend without rebuilding every application.

Bugs in the log shipper has the possibility to take down your app, you never want this. Also think about the unnecessary security surface this adds, especially when multiplied by M above. (log4j anyone?)

When the log server is unavailable or saturated you still want those file or console logs available for last resort debugging.

Shutting down the app risks loosing last minute of unshipped logs, unless you require slow graceful shutdowns, which is never a good idea. Think about crashes, where you need logs the most.

Use sidecar and similar strategies instead. You are most likely also running some third party application elsewhere, where you don't have the option to rebuild with a new logger, so you need this type of knowledge anyway.

It’s frustrating that this is necessary but it definitely makes sense, and I will probably add it to several projects soon.

Thanks for your hard work!

Thank you! Please raise any issues if you encounter any, or have any feature suggestions!

Makes me think of the big tent philosophy of Grafana Alloy for OTEL

Where does log layer fit into the broader ecosystem?

https://github.com/grafana/alloy

This new version of LogLayer was actually created because someone requested OTEL support. Unfortunately I'm not too knowledgable about the OTEL ecosystem (outside of using / implementing the DataDog APM tooling) to know where the integration points could be on a broader-basis.

Although I lack experience, this version of LogLayer allows people to build their own transports / plugins that someone with experience and familiarity in this space can build a LogLayer integration.

That being said, I'll definitely start reading more into it with the link you provided and any other resources would be appreciated!

Grafana LGTM + Alloy is a great jumping off point

Thanks. I spent a bit of time looking at otel integrations and added support!

https://loglayer.dev/transports/opentelemetry.html

https://loglayer.dev/plugins/opentelemetry.html

Situation: there are 14 competing JS logging APIs.

https://xkcd.com/927

Reply on news.ycombinator.com