Show HN: Piping logs, visualizing in a web app – just suffix "| npx logscreen" | Hacker News Reader

Show HN: Piping logs, visualizing in a web app – just suffix "| npx logscreen"

github.com

128 pointssoorajsanker79 comments

I'm not super familiar with express but this seems to be listening on external interfaces as well:

https://github.com/soorajshankar/logScreen/blob/f8b29aaef428...

So this will be sharing your logs with the world if you run this outside of a trusted network. (And personally I avoid trusting any networks)

Are you talking about the difference between 0.0.0.0 and 127.0.0.1 /localhost (loopback address)? If so It should be a tiny thing to change to implement I can add that on the next update- I just went with the express JS defaults to keep things simple:)

Or I can even make a flag to configure this :thinking

`app.listen` uses the same signature as Node's `server.listen`. You can specify the `host` [0] or else it defaults to `0.0.0.0`. This means that when you run this service, it will be accessible to anyone that can reach the server.

Typically, services allow you set the host address by configuration. This way you can set it to `127.0.0.1` so that it is only available locally.

[0]: https://nodejs.org/api/net.html#serverlistenport-host-backlo...

If you do make it configurable, it's still a good idea to make the default `127.0.0.1` for the reasons @ggpsv mentioned. Since the link automatically opens at `http://localhost:PORT` anyway, it's an easy security win.

It does, but express doesn't tell it directly in the docs.

Instead http://expressjs.com/en/4x/api.html#app.listen sends you to https://nodejs.org/api/http.html#http_server_listen which also won't give the answer but sends you to https://nodejs.org/api/net.html#serverlisten which at some points finally documents that default is 0.0.0.0.

Welcome to the JavaScript ecosystem :/

I wouldn't exactly call this a failure of the JS ecosystem. The Express documentation you've linked to mentions that the host is the (optional) second argument to the listen function:

> app.listen([port[, host[, backlog]]][, callback])

You literally just have to run:

  app.listen(3000, '127.0.0.1')

If you don't trust the network you're on you should just be explicit about which IP address you'd like to listen on.

That's a sensible default. It would be super frustrating if by default the web server you built couldn't be accessed by anyone else in your network.

Congrats on launching! Its interesting to see that ChatGPT is able to build something I've been working on (https://logdy.dev) for last two months. I guess new era is around the corner then.

Tbf your tool looks 110% more useful and well built than a simple single screen showing the logs with an in-browser search. Keep at it!

Thank you! I have a full roadmap of features to deliver :-)

Your landing page looks fantastic, just wanted to throw that out (before ChatGPT starts making that, too)

Thanks!

I was looking for something like this for a while, and it could be just the landing page, but I much prefer a tool that's been built slowly with thought and consideration (which is evident from your landing page).

How does your tool handle semi-permanent usage or very long logs? For instance, I would run this to view logs from an ssh server that's been up for 6 months, when systemd has rotated the log file etc.

Why would you watch logs for more than 6 months?

Do you filter them? What does the system you are watching do?

for instance, running a nextcloud instance through docker compose, I'd like to have a lightweight web ui for browsing the server logs. not just for 15 minutes but for longer periods of time.

on an hpc cluster, I want to want logs from the slurmctld forever essentially without having to login and grep through the logs.

there's a bunch of online services for this, there's ELK stack, there's grep, and few things sit in between. hence my question

then the problem is not viewing the logs but storing them, maybe compressing, and then retrieving. If that's not a problem (size) then with some cron magic you should be able to store the logs from specific locations into a single one so rotation doesn't eat them and then just view using a log viewer of your choice.

Thanks for your kind words. I have a feature onnthe roadmap which is very close to delivery that will allow you to load a big log file and browse it through the UI. As long as you can load the content from disk or remotely, Logdy will be able to handle that. The aim isnto provide DataDog like experience but in a portable and temporary way, that allows you to set it up quickly, check what you need, tear down and move on. All without much hassle or costs for sending the logs to a remote service.

This is a great tool too and echoes task teleportation, which is popping up more and more.

There’s a new era, but one in which you will be even more powerful. LLMs will improve in pumping out code embodiments… but it was your human creativity borne out of operations frustrations, coupled with your care to do something about it, that manifested your product.

Thank you for this comment, you got a point and this is indeed a good thing that's happening!

Please keep working on this! I started a similar tool because I desperately needed it, but haven’t found the time to maintain it. https://github.com/corytheboyd-smartsheet/json-log-explorer

Having something like the log explorer in DataDog for local development is so insanely useful

That was the idea exactly: I wanted to have DD log explorer locally that works insanely fast to be coupled within development process. As much as I love terminal, its just so limiting when it comes to browsing and filtering through more content.

Absolutely agreed, and glad someone else is out there fighting the good fight :)

> As much as I love terminal, its just so limiting when it comes to browsing and filtering through more content.

What terminal-based tools are you thinking of when you say this? Just tail/grep/less?

that and more magic of sed, awk, jq and similar. I know you can do real magic in there but still, I believe it's easier to learn GUI than vim shortcuts or sophisticated jq/awk formulas. Still, I admire terminal for being around for so long and follow it as an example when building Logdy.

Is there a benefit over

    command | vim -

I love doing that, because then I get all the power of Vim to look at the output. Search, regex search, sorting, bookmarks, counting, macros ...

I started to write output that is indented, so I can use Vim's folding to comfortably look at complex data.

For example if you have entries like this:

    2023-04-12 17:22 Sighted and UFO
        Location: 27.1742424 12.137234
        Temperature: 24°C
        Observer: Joe
        Actually it turned out to be a dove.

And you press zM in Vim, they all turn into:

    2023-04-12 17:22 Sighted and UFO
        + 4 lines

So you have a nice overview and can open individual entries by putting the curser on them and hit zo.

And while folded, you still have all the power of Vim to search within them. Regex search. Count by any measure you like etc etc. Actually, you can do anything, as Vim allows you to execute arbitrary vim commands, vim macros or shell commands on your buffer.

Wanna know how many doves typically get mistaken for UFOs around New York grouped by month? Write a script for it and it will be forever at your fingertips.

I think this is more designed for streamed output rather than viewing files. Vim doesn't seem to support that, in fact it just says "Vim: Reading from stdin..." forever.

Hmm.. I guess you could write a little script (2 lines?), let's call it "vimlive" which reads from stdin, streams to a temp file and opens that temp file in vim.

Then with:

    command | vimlive

You get vim with the current state of the data and you can update it with :e

You can have vim constantly refresh a buffer if it's modified externally, like this turns your buffer into a kinda-sorta-version of `tail -f`:

    :set autoread | au CursorHold * checktime | call feedkeys("G")

So if it's already a file, just open it with vim; if it's a STDOUT/STDERR stream, redirect/`tee` to a file and open that.

Also it’s not as snappy when you cross the 1GB limit, although it does work, mostly.

With a quick look at the code I'm pretty sure this WebUI also wouldn't be snappy along before that.

try

    vim filename

lmao

> Interactive Interface: Search, filter, and navigate through logs easily. // TODO

This is exactly why you would show logs in the browser though— this is not a nice to have, it is THE core feature. The browser is perfect for building the complex UI necessary for filtering and making sense of log streams.

That’s probably why it’s on their TODO list

Ahh actually you can do search through the logs in real-time, I forgot to split that and remove TODO from the readme. Complex filtering based on the timestamp is yet to implement.

Found the implementation: https://github.com/soorajshankar/logScreen/blob/f8b29aaef428...

"This tool has been generated entirely with ChatGPT and hence it is also a demonstration of how useful products can be developed with collaborative reiteration with GPTs."

Any chance you could share a link to the ChatGPT conversation, or a copy of the prompt you used?

I build little tools like this with ChatGPT all the time, it's one of my favourite use-cases for it. I often have an idea for a tool that wouldn't justify an hour of development work but is absolutely worth spending 5 minutes iterating on with ChatGPT.

Nice, will give it a try!

There is a similar tool that I've been using for years: https://github.com/mthenw/frontail/

This is handy stuff, thanks for sharing. We are working on transporting tasks back and forth from TUI<>Web<>App<>VR ; this is a great and straightforward example of that.

I use VSCode CoPilot constantly as simple augmentation and typing-saving — not for wholesale creation of code. I’ve seen my pre-novice daughter use ChatGPT and the results were mixed (good in some ways, horrible in others).

Can you elaborate more on your ChatGPT/LLM working style for this?

Sure thing! The collaboration with ChatGPT was like having a friendly coding buddy. I threw in ideas, got feedback, and we built this tool together. We chatted about names, features, and even how to post it. It's like having an AI pal who's great at brainstorming and coding. Fun times!

^ asked Chat GPT to generate this from my earlier prompts :)

Some other notes, I am a developer who love to make usable products, and these years I learned the usability is most important especially when you are developing in the early stages of a product. So I fed some ideas, asked suggestions, gave some tech stacks (for example, I asked chatGPT to not go with webpack, instead simply load React from CDN for quick iteration. and suggested memoisation on React to render things faster). So process wise similar to any other product development, but it's like having someone listen to your feedback and instructions and do the work- I would like to call collaboration!

Thanks for those details!

Looks neat. Don’t listen to the haters

This is a solution looking for a problem. The browser is a horrible place for viewing text logs. Companies like Splunk and Datadog put countless hours into trying to make it work and it's never not-painful to view logs in a browser.

What are the main specific issues that you felt difficult in viewing logs on a browser? imagine there is better visualisation, parsing, highlighting, filtering, collapse etc.? I feel like the terminal is where I spend more time looking for the relevant parts from the logs.

Some CLI viewers for logs are listed on https://github.com/dloss/klp, as alternatives to my own little tool "klp".

What's better? I ask out of ignorance, but personally I think Datadog is amazing for viewing logs.

This is pretty neat! I did brew upgrade | npx logscreen as the first thing I could think of that would generate logs to play with this with.

Good idea. Thank you for sharing.

One thing I've always wanted was simple filters on log lines in a GUI that let me collapse tags in the log line. I created a proof of concept that showed logs as a tree.

Or I can say I want logs from the point where key=value and until method x.

A bit like a debugger but for logs. Show logs that are between these facts.

> showed logs as a tree.

Yes, that is exactly what I want as well - a tree-like structure where you can trace events across multiple layers of abstraction as well as across systems (so both horizontal and vertical relations of events, so to speak).

Some infra systems output syslog loglines and there's no changing that, and I still want to capture and relate these log messages to other events (incl. high-level business transactions / processes).

Any way you could share your PoC? :) very interested.

Thanks for your lovely reply.

I don't have the code anymore might have been called "logtree", it was a HTML file and very small, it would parse dot "." separated lines such as (user.connected) and arrange them in a tree with indentation, if you clicked the dot syntax it would highlight in yellow other instances of that bucket of entries later in the log file.

I think you want elegant navigation between linear sequences (next, previous) of log lines that are apart in time, but related.

You might want to see all events of the internals of two different components in the tree structure.

Need to process log lines into a graph and then provide quick navigation for them.

It is indeed horizontal and vertical.

This would be very useful to anti-abuse measures if you could trace to the cause of something and cancel all thing caused by it.

Thank you for helping me structure what I want, yes, exactly. (And noted re: your PoC, ok!)

> You might want to see all events of the internals of two different components in the tree structure.

> Need to process log lines into a graph and then provide quick navigation for them.

Yes, looks like it, indeed. Yeah.

If interested, this is how I'd hastily summarise my problem statement - it is I think a rather generic recurring problem and analysis pattern, not unique to any company (context: goal is to drive through this initiative as part of overall software architecture improvement needs, and bringing business maturity (as per CMM model) up in our particular case):

- need to introduce and enforce structured event hierarchy (think: each high level business transaction (e.g. user signs document) is a parent which retains all related child events across all layers of abstraction (ideally down to DB connection and file handles, internal API endpoints hit, block storage i/o errors, etc.)

- problem: while for own software Backend-generated events we can enforce and manage this, some infra pieces emit raw text syslog (sometimes even syslog message portion (log format) cannot be fully controlled) and do not contain cross-component technical identifiers for determining relations to events

- there are solutions including ML solutions (of course)

- there exist interesting papers to lose productivity and/or conduct deeper research

- what do, how to balance, how to incrementally define, introduce into culture, incrementally realistically establish this (etc.)

Some interesting papers for approaching the complicated part (assuming defining event ontology and hierarchy and overall semantics is the easy part) - the "relating unstructured logs to events and embedding them into graphs, and aligning and training everyone to analyse and work with "event trees" across multiple teams (but yes, ops, integration support, etc. - I mean is that not the dream, right...) do exist[1][2][3][4][5], and some interesting thoughts and many lofty ambitious questions[6]. Trying to gain insight into best practices here. Meanwhile, maybe links will look interesting :) (e.g. that 1849-2023 IEEE standard - just found it today...)

[1] LogTree: A Framework for Generating System Events from Raw Textual Logs https://www.researchgate.net/publication/220766390_LogTree_A... (https://ltangt.github.io/papers/icdm2010-logtree.pdf)

[2] Structural Event Detection from Log Messages https://faculty.ist.psu.edu/jessieli/Publications/2017-KDD-l... (this one may look bewildering at first but the math eqs are actually ~legible / straightforward from quick glance, even; also, has nice coverage of related work) (overall nice, might look smarter than appears, but not sure; see MCMC approach etc. in section 5.)

[3] golang package https://pkg.go.dev/source.monogon.dev/metropolis/pkg/logtree - why i mention - see KLogParser func (https://pkg.go.dev/source.monogon.dev/metropolis/pkg/logtree...) (but also see limitations) (did not review in detail, maybe nothing useful)

[4] 1849-2023 - IEEE Standard for eXtensible Event Stream (XES) for Achieving Interoperability in Event Logs and Event Streams https://www.xes-standard.org/

[5] related to above (supports standard) https://pm4py.fit.fraunhofer.de/documentation -> see example of ontology: Object-Centric Event Logs

[6] a structural approach to the problem of structure... use existing frameworks and languages like BPML to create or use existing event type ontologies from BPMN models; use a proper structured thought-through ontology for this, and then sort out that pesky "relating low level infra raw syslogs with none/few TIDs to some node on a graph" problem

Anyway, thanks for making through my rant! Writing it in part because on the off chance someone bites and has insight - I'd be very interested to hear more; and partially just to share those links above because those may tickle your (Sam's fancy) :) thanks for reading!

Neat, thats a pretty good use of a web app.

How is it handled when I want to view logs that are too large to fit in memory?

Install more memory

Download more RAM

Depends on the size, for me it worked well with decent size JSON log lines, but if it is too much that doesn't fit on the browser memory, can think of index storage of sqlite in the future. Search and filtering can be then loaded to the server + sqlite setup.

Wanted to give kudos. This is one of those solutions that “just works” and was really nice to see. Well done!

Did you know AWStats (first released 2003) could accept streaming logs and produce nice dashboards? https://awstats.sourceforge.io/

This is very neat. It would be super cool if we could extent it to multiple processes. From the Looks of it , it should be very much possible.

Very tempted! Installation instructions?

Simply suffix with any unix commands that emits stdout, ` | npx logscreen` (need npx though)

`npx` is a tool that ships alongside `npm` (alongside `node`) that either finds a local installation or performs a just-in-time download, and then runs the default "bin" entrypoint for the package.

Newer versions of `npx` (last few years) will prompt the user before installing a package. Older versions did not. Note that installing the package can run arbitrary scripts as a side effect with at least the level of permission of the current user, so there is implicit trust required of the author, and the authors of all transitive dependencies.

Thanks. Unfamiliar with the Node ecosystem, that escaped me

> This tool has been generated entirely with ChatGPT and hence it is also a demonstration of how useful products can be developed with collaborative reiteration with GPTs.

All you need to know.

I was going to ask about the strange combination of package.json and inlined cdn js dependencies (especially strange because it's a local server)

Suppose that answers it.

Haha yeah, I wanted to frame this as a single file tool, and hence all the quick wins. But later moved the react app into a separate file because I was not getting the syntax highlights and all.

Definitely this can be made better with proper packaging and tooling! As I mentioned, more than the idea, implementation was an experiment on the ability of collaborative iterations with GPTs- it was fun!

Oh- how much manual intervention was there? It was not entirely generated?

Yeah, there are better ways to do this, I wonder if it's purely an excersize in cgpt useages.

True, This was also a personal experiment in understanding the level of collaboration that someone can do with GPTs during the product iterations.

> All you need to know.

For those of us not in the know, what does that mean?

> For those of us not in the know, what does that mean?

I reckon that some people are either critical of LLMs due to the data that they're trained on (concerns about the legality of using the outputs, or the ethics of taking open source projects with a variety of licenses and how transformative or not the output might be), or question the quality of the code that they might output. Oh, also there's hype cycles and right now people at large might overestimate the capabilities of what LLMs can or cannot do well, which can be tiring, as seeing constant posts about crypto was (where we had a solution in search of a problem).

In my eyes, they can be used in conjunction with something like IntelliSense and IDEs to solve select problems (ones with solutions that already exist out there) more quickly, as well as save time on some boilerplate, as long as you still validate the outputs and actually check if everything works. Nowadays I'm using GitHub Copilot and the user experience is mostly okay, I'd say it actually lets me write code faster, at the expense of occasionally getting things pretty wrong, but is still a net positive, other considerations aside.

In that case, I'd quote the guidelines to GGP poster:

> Please don't fulminate. Please don't sneer, including at the rest of the community.

https://news.ycombinator.com/newsguidelines.html

If it works, I don't see anything wrong with it. It looks good. Hating on projects that disclose their GPT usage will only incentivize them to not disclose in the future.

As much as I love to hate on chatgpt engineer, the code is definitely okay; not much difference from a project you would write quickly in a single weekend.

The first thing that struck me when I looked at some of the code was:

    <div className="flex items-start border-b border-gray-300 py-2 hover:bg-blue-100 cursor-pointer transition-all"

I'm not saying that humans never come up with this kind of crap, but it feels like it's going to be a LOT more common the more we resort to AI. I wonder if "... and make the code easy-to-read so that it's maintainable in future" is ever included in prompts for LLM-generated code.

That's just Tailwind and exactly how my HTML looks as well.

I'm a clean code guy and tailwind made my code look terrible. But I gave it a chance because I heared so many positive things about it. Now I'm happy with it. I don't have to jump around from HTML to CSS and back to HTML.

that's the horror of Tailwind for you, which is different from the horror of AI

Software development is horror all the way down. The sooner you embrace the horror, the easier life gets.

That is just tailwind CSS, one the most common CSS libs out there I would think. Your comment is actually a lot more telling of something as opposed to this AI written project though.

> Features

    Web-Based Log Viewer: Get a cleaner and more organized view of command outputs.
    Real-Time Updates: Logs are displayed in real-time as the command executes.
    Interactive Interface: Search, filter, and navigate through logs easily. // TODO

ChatGPT has its limits // TODO

Reply on news.ycombinator.com