Ask HN: what software do you use for analyzing apache logs?
What software do you use for analyzing apache logs? I would prefer open source ones that will work on Linux, although this is not a deal breaker.
I tried Google Analytics too, but found it kind of annoying in that it's sort of a pain to get the information I want.
Edit: Also if you could like to play with real-time analysis to get detailed information about how your users are interacting with your web site please feel free to write me at antirez -at- gmail ~dot~ com for a free invitation for http://lloogg.com
I used it from 2004-2005 for clicktrails... great idea and implementation. Graphviz makes everything better (corollary: anything that needs Graphviz is interesting enough to get better)
Google Analytics and/or Visitors are about all I would think a person needs. And now that you have lloogg.com out there, it looks like clicktrails are covered in a GA-style interface. Kick ass! Great idea, again, best of luck.
I haven't run into many websites that couldn't be improved by clickpath analysis and refocusing navigation on the ways people actually use a site -- optimizing for the common case, in other words. It baffles me that any site would fail to do so. Now they really have no excuse.
LLOOGG is still pretty raw and we are developing the "filters" part of the UI, but still it seems pretty useful to check what's really happening in a web site. We have a lot of success stories of people using LLOOGG for some week and then modifying the site structure to optimize the user experience.
Thanks for the comment.
Webalizer's ok, but it's missing a bunch of features like sane user agent string parsing (to give an overview of the browsers accessing your site). It also displays most of the stats by page hits (such as country and user agent) rather than "visits" or unique IPs, which I think is a better way to group.
I've been pretty happy with Google Analytics so far - it has a ton of options for sorting and grouping data (like viewing users' paths through the links on the site) and good IP geolocation. Plus, the JavaScript tracker gives you stats on visitors' screen resolution, which can be handy. On the downside, all our data is belong to Google.
It's based on this perl module: http://cpan.uwinnipeg.ca/~peterhi/Apache-LogRegex
To manage log files, use cronolog
edit: I use cronolog to break up the logfiles daily and then I run a python script (which uses apachelog to handle the nasty parsing) to create a summary dictionary of parameters from that day. For example:
{num_unique_ips:140, num_pageviews:532, ...}
I pickle that dictionary and save it as a file. So every day has a raw log file and a "summary dictionary" file. To make plots I go through the summary files and unpickle them to extract the quantities of interest.
sudo awk '{print $11}' access_log | grep -v 'yourowndomain.com' | grep -v 'bloglines.com' | grep -v '"-"' | grep -v 'feedburner.com' | sort | uniq -c | sort -rn | head -20
along with cronolog and a bunch of custom Python scripts to autogenerate webalizer configs.