Show HN: My weekend project, realtime log viewer in Flask | Hacker News Reader

Show HN: My weekend project, realtime log viewer in Flask

logger.conceptis.org

58 pointsllambda26 comments

So if anyone's wondering how this works, it's actually quite simple thanks to juggernaut. I used Flask to build the web app but because it's basically just serving up a page that really wasn't necessary. You could actually write this all with something like gevent, however Flask lends itself to extensibility and you could certainly use this as a component of a larger app.

The primary functionality goes like this: Essentially I wrote a small script to read off a log file, that runs concurrently as a greenlet via gevent. Anything it reads is pushed to juggernaut which uses socketio to publish that along a channel. The frontend reads off that channel, pushing in realtime.

I actually spent more time styling it than I did coding it. The VT terminal style is very intentional and I'm pretty happy with the overall result of the look (hey, CSS3 is nice!). I also included a couple of alternate color schemes: blue and orange.

Hope you like it! :)

I understand Flask can not do socketio because it is WSGI app. Any idea if socketio will be supported by WSGI at all?

In addition to this, I recently discovered that Flask doesn't support Transfer-Encoding: chunked (which is part of the HTTP/1.1 spec), and doesn't fail gracefully either (when receiving a chunked request). Then I discussed it with Armin and realized that the WSGI spec prevents Flask from doing either. (See also http://www.b-list.org/weblog/2009/aug/10/wsgi/)

Time to fork WSGI?

ETA: Upon further investigation, there's a big discussion out there on what to replace WSGI with, here's one entry point: http://dirtsimple.org/2011/08/is-wsgi-lite-library-or-protoc...

There was/is an attempt to do this with gevent via gevent-socketio. However that project seems to have stalled and my last attempt to use it, a couple of weeks ago, did not have good results. Armin Ronacher, the author of Flask, actually used juggernaut in a sample application which led me to experiment with it.

Cool. And not to take away from it all, but ...

You can do the same or better with multitail locally tailing a remote tail over ssh, and you get multitail's color schemes for various log types like mail or httpd.

    multitail -s 2 -sn 2,2 -C \
    -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/nginx/a*" \
    -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/nginx/e*" \
    -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/uwsgi/*" \
    -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/ufw.log"

That's pretty cool, hopefully we will have better support for multiple logs soon

What is really bugging me as a programmer: We live in the 21st century and people still include user-supplied data in their pages as if Cross-Site Scripting (XSS) is totally unknown.

I get a bad feeling when these simple attacks are possible; because this shows that there are fundamental flaws in some people's coding styles. I know this sounds harsh, and I recognize it is llamdba's weekend project, but these flaws show neglection of fundamentals in handling data.

Apart from this, this live log is very entertaining!

You're absolutely right, I overlooked the fact that the log data wasn't being escaped (something that I had realized in a previous chat application I had made last week). A quick fix is to use flask's built-in escape function, something like: line = escape(line). The server is now escaping lines off the log.

Thank you for the entertainment, unknown people of HN.

X.X.X.X - - [27/Dec/2011:00:26:07 +0000] "GET /hey-i-can-send-secret-messages HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2"

Y.Y.Y.Y - - [27/Dec/2011:00:26:19 +0000] "GET /I%20can%20recieve%20secret%20messages%20too HTTP/1.1" 404 211 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0"

X.X.X.X - - [27/Dec/2011:00:26:25 +0000] "GET /hey-i-can-send-secret-messages/maybe-al-qaeda-could-use-this HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2"

X.X.X.X - - [27/Dec/2011:00:26:50 +0000] "GET /ooh-you-can-recieve-them? HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2"

Z.Z.Z.Z - - [27/Dec/2011:00:27:05 +0000] "GET /they+are+already+using+it HTTP/1.1" 404 211 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0"

X.X.X.X - - [27/Dec/2011:00:27:34 +0000] "GET /hello-decatur! HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2"

Y.Y.Y.Y - - [27/Dec/2011:00:27:56 +0000] "GET /this%20is%20alqaeda%20checking%20in...%20Hussain%20do%20you%20copy??? HTTP/1.1" 404 211 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0"

X.X.X.X - - [27/Dec/2011:00:28:15 +0000] "GET /200-hussain-found HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2"

I coudn't respond to hussain :(

I can immediately envision some use cases for this RTM at the workplace, yet our log file throughput is extremely high, so I would have to incorporate some type of throttling mechanism, Great Project...

Nice demo, thanks for enlightening me of juggernaut's existence! If I were to be picky, it's hard to read anything because the scroll position moves to the end even if you explicitly control it, which it shouldn't. Plus, there's this reliability problem that between your Flask code that serves the last 100 lines and the realtime communication, you can miss some entries. And it's a bit inefficient such as loading the whole log file just to grab the last 100 lines. Perhaps this comment should be on GitHub...

Yeah it's definitely a little rough around the edges. If you're interested, I'd be more than happy to merge your suggested changes via a pull request on GitHub.

I was planning on writing something like this in the next few weeks for my Flask app that runs on the Rascal (see profile for details), but now I won't have to.

Thanks, Max.

You're most welcome. Please feel free to contact me if you have questions about setting it up. Also if there's changes you'd like to make that would benefit the application in general, I'll be happy to accept pull requests on GitHub.

Roger that. I'll let you know if I come up with anything good.

It's poetic. It's like walking into a room filled with mirrors.

Why is it that Flask is highlighted when juggernaut, node.js and redis do the realtime communication? :-/

Because it's not built in a JS framework such as Node (it just uses those things as a secondary service). Which up until recently was kind of a pain. Juggernaut makes life easier and as a result there has been an increase in realtime Flask applications which is pretty cool if you're a Python web dev.

It's fun to watch. You can communicate with your browser agent.

Response 500 :(. I was having fun chatting with GET requests..

XSS By bflesch@bf-itservice.de :) nice work

https://github.com/maxcountryman/logmon/issues/1

I was playing a bit with the app and it seems some attack vectors went through. I really didn't want to mess up your experience of this POC and see that you've already filed a report on github, that's very impressive. Thanks. :-)

Seems to have some XSS vulnerabilities

Yeah we had the data just return instead of escaping the value.

Reply on news.ycombinator.com