GoAccess – Visual Web Log Analyzer | Hacker News Reader

GoAccess – Visual Web Log Analyzer

goaccess.io

Remind me of this cool project called netdata[0], also written in C.

[0] - https://github.com/firehol/netdata

I've used both. GoAccess tends to be more useful to me when I have a set of log files and want to track down something specific to the web server config. That's particularly handy when working on a server I'm not already familiar with, or when I have logs but no access to the server, since I don't have to run GoAccess on the server itself. It's also the only IIS log analyzer I bother using.

netdata is more useful for tracking live stats across live servers that I'm actively maintaining (especially multiple servers), and pulls a broader set of information from more sources, which is handier for tracking down intermittent networking issues.

They can both do a little bit of each other's job, which is nice. I tend to use GoAccess more often but think they're both pretty good tools for visualizing server data.

I am having the hardest time getting goaccess to work with the IIS 8.0 W3C logs. It keeps on complaining about the date / time format. What is your secret sauce?

Try uncommenting the relevant date/time/log formats from your goaccess.conf. Then run goaccess -f access.log -p /path/goaccess.conf

Or https://gist.github.com/soheilpro/a34957550b1bd7d42be2

This looks quite useful, and I think I'll be using it. I'm impressed by it being written in C... Especially because it can also run on the browser! Also, the default storage being Hash Tables. You don't see many modern projects like this! Actually feel like diving into the code.

I used it and I can tell you that it works exactly as advertised :).

I've been using GoAccess for a year now as a traffic analysis tool on my blog. It's been great. Easy to use, and great for real-time analysis using the command-line interface.

This looks fantastic. Any idea if it supports custom NGINX log formats?

    log_format main '<$host> $remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

Yes, it supports custom log formatting but you'll likely have to write a bit of parsing code in the config, see docs here: https://goaccess.io/man#custom-log

There are also a lot of log formatting q&a in the github issues (both closed & open) if you're using a popular format.

Does it similar query and search capabilities like Splunk or Kibana. Is it a full stack log system or just a UI?

It appears to specifically parse web server logs. Unique visitors, counts of 404 pages, etc.

Similar (in purpose) to "Awstats" or "Webalizer" that you find on almost every shared hosting account.

It seems is still in the works... https://github.com/allinurl/goaccess/issues/117

Can I use this to visualize custom data?

for example, can I push stock prices to this platform and visualize them in real time?

It's a web log analysis tool, not a general data viz platform. This may be more what you're looking for though: http://www.cyclotron.io

just the other day - in lieu with AS ipblock info and 'rgxg cidr' - helped me see at a single glance (3xx to 2xx relative count) some redirect deficiencies of a platforms http image fetchers. As you can pipe to it, it complements the other text utilities filtering logs very well.

How is the performance when dealing with large log files? I get about 300 million pageviews a month

See #Performance in:

https://goaccess.io/faq

> A dataset of about 52M hits (12GB size) is parsed in 20 mins (in-memory), 60 mins (on-disk storage).

My logs are about 500MB a week and it takes less than a minute to process on my 2 core / sata server.

Currently using this on one of our clients sites and I love it! More tools like this I say!

Quite useful. Would be nice if it could be fed from piped input too.

It works just fine with piped input actually, I use it regularly with zcat.

Indeed, most of the GoAccess docs around pipes provide examples for using them to backfill missing expected features, like filtering by date range, file/MIME type, status code, etc.

I use `find` to choose specific months and pipe the log files into goaccess (if you have lots of logs, that saves time instead of piping everything). Assuming the logs are in a bunch of gzipped files:

  find my_logs/ -name "*.gz" | grep '2016/10|2016/11' | xargs -I file zcat file | goaccess

Can I use this in Docker/Kubernetes?

Yes. Just mount the logs into the container.

Actually, the problem is that Docker logs to stdout rather than to logfile(s)

That's what Docker's logging drivers are for. It does so by default, but it doesn't have to.

https://docs.docker.com/engine/admin/logging/overview/

Should I have log files if I host a static (very high traffic) site? Everything is static. I've turned off the logging, no idea what performance gain it has, if any, but is there a reason I should write then in file? I don't see any need.

Probably should. The logs have some pretty interesting details in them.

http://attrition.org/postal/logs/2006/logs.001.html

NSFW. Should've guessed due to it being attrition.org.

Reply on news.ycombinator.com