randolph_carter

I enjoy http://bit-player.org - it's a blog by Brian Hayes, who writes on computing science for American Scientist magazine (but is currently on sabbatical from the magazine, hopefully not from the blog).

Also not wanting to sound like a mouthpiece, but choice of log management system is a very interesting topic to me... scalability is only one concern and may not trump everything. We actually looked at both LogLogic and Sensage during our initial evaluation both of which scaled better than Splunk did at the time. My memory of the evaluations is dim (2+ years ago), but I recall that both products seemed more brittle in our environment (education), where we have very little control over incoming log formats due to somewhat chaotic organizational structure and infrastructure. Splunk seemed more able to accept unstructured or unexpected data without complaint and to deal with the parsing at search time rather than index time, and was a more natural fit. The other decision points/tradeoffs we ended up making were a) no additional licensing cost for Splunk agents, b) prioritization of real-time perusal for troubleshooting purposes over long-term reporting and canned reports. Sensage excels at the latter since it amortizes the cost of reporting by caching periodic results, and provides a large set of prebuilt reports, if I recall correctly. Splunk's "summary indexing" serves sort of the same purpose as the cachign feature of Sensage and has been improved in v4.0. And c) no additional cost for users (we have over 50 people looking at data indexed by Splunk). The "search" metaphor is also very easy for people to grasp which was an added benefit of the Splunk GUI.

My sense has been that at least the low end of the market is increasingly preferring a search-oriented log management architecture versus a database-backed, query-heavy architecture because organizations are familiar with the search metaphor, the overhead of managing a search index is less than the cost of database administration, and the unsophisticated use case (i.e., free-text search rather than advanced query syntax) is increasingly common. Smaller organizations also rarely have the maturity to deal with logging systematically since it requires a pretty systematic approach to infrastructure. That said, the tradeoffs we made may or may not apply in a terabytes-per-day environment, I unfortunately can't speak to it directly.

Thanks for the additional context... like one of the other posters mentioned, I think once your target organization has evolved beyond the capabilities of the existing products in this space, the knowledge and analytics built-in to the product become the product differentiator. Scalability is assumed. For known problems, the real value of a log analysis tool is in the knowledgebase you buy with the product and the canned reports you can get out of it; that's why there are specialized SIM/SIEM products out there. If it is an unknown problem, your customer will end up generating the knowledge base themselves anyway. Unfortunately, many of your customers may not even know what their problems are, and will expect you to sell them an "Easy button" solution...

All that said, good luck in your endeavors!

To corroborate this, it has not been our experience that Splunk doesn't scale. We have a ~40GB/day environment and according to support can scale that to ~100GB/day on a single quad-core server with SAN back-end, license permitting. The lag between reception and indexing is negligible; we actually have some developers looking at their own debug logs in it.

Unfortunately, I don't work in a TB/day environment (though that would be fun), so I can't comment on that directly, but our experience with Splunk has been positive at this scale and I see no reason so far why a distributed installation with multiple 100GB/day nodes would change that. The only customer complaints we have are basically that long-term searches don't return instantly, and that's primarily a factor of I/O speed and can be resolved by setting up summary indexing (which requires forethought about what data you're interested in... therefore, it very rarely happens in my organization).

Out of curiosity, can you be more specific on what scalability, analytics, and ease-of-use improvements LogSearch offers vs. Splunk in particular? We use Splunk internally and have not found it to be significantly lacking in these areas but are always open to alternatives. Additionally, what is the advantage of relegating log management to a cloud application? Typically log data is relatively sensitive; do you provide assurances that the data won't be misused, lost, repurposed, etc.?