Hacker News Reader

Comment by packetized | Hacker News Reader

This is the latest in a long string of irritations that will have me looking for a replacement logging datastore post-haste. I've already replaced Logstash with Heka; I guess ES is next.

View on HN

Good luck with that searching, and remember to share with HN.

I find ElasticSearch somewhat brittle (I need to restart it every few weeks or so, because it stops accepting any data or queries), and I really do want to replace it, since it's a memory hog (no data, and it already needs 230MB RAM), but I haven't found any sensible log storage yet. All I got is this document searching engine.

230MB of Heap is very small for most ES usages, and is likely the reason for you needing to restart it often. I run some very large clusters and even with older versions of ES have seen them run billions of indexing and query operations without problems for 4-6 months without any restarts.

> 230MB of Heap is very small for most ES usages

230MB RSS, and this is not "ES usage", it's just after starting it with no data whatsoever. I've seen databases which use less memory than that when working.

> I run some very large clusters [...] without problems for 4-6 months without any restarts.

Well, good for you. Being a sysadmin, I understand very well that my ES server may be a specimen combining bad kernel compilation, broken JVM version, unstable ES release, and ground water emanating bad energy, so I don't complain much. Still, I would be very, very happy to see any alternative for storing JSONified logs.

Despite what they recommend, I'd move to G1GC over CMS. Did wonders for our stability. This is on ES2.1.0 & OpenJDK 1.8.0-u80, IIRC.

That was my experience exactly. The value of being able to search through ours logs was immediately apparent, so when it kept tanking every couple weeks it was fairly easy to talk my boss into paying a tiny bit more for Loggly. AFAICT Loggly is the ELK stack with a different theme applied to the UI, and I don't have to worry that it's down when I need it.

The preceding has been an unpaid endorsement for Loggly.

I would prefer much more to have something hosted by me and being open source at the same time (I'm looking at you, Splunk). I'm desperate enough to write such thing myself (and I'm already doing some research on indexing semi-structured data), even if it would turn out not-too-scalable (I only have ~1.5GB/day of uncompressed logs), but I'm scared by the need of tolerance for abrupt shutdowns and hitting a free disk space limit, i.e., database recoverability.

Hey, I looked at Splunk in the last few days as well, but it seems really expensive once you want to get beyond 500MB a day. They don't have an OSS "community" version, do they (?) because I totally feel you on that.

Basically I need ELK, but I need it to be a lot more stable and documented better.

Loggly is a five digit monthly bill at our present logging volume. No thanks.

What are you using? I'd rather run it myself, but like I said ELK was a bit too bleeding edge for me in the stability dept.

Heka for ingestion/parsing/message routing, RabbitMQ as the queuing/delivery mechanism, Elasticsearch as datastore, Grafana for visualization.

I suggest taking a look at the logz.io - it's essentially ELK as a service. We take care of many problems ES has and also have great features extra such as Alerting, Apps and Insights. I'm one of the core developers there so I'm fluent in pricing but last time I checked it's wasn't expensive and definitely worth your time and hosting costs. One the the great things about it is that you can simply migrate your dashboard and just ship to our Url.

I'm looking at Graylog right now.

Which version? This reason I ask is that 2.x is a lot more stable for me than previous versions. I have to restart much less often and I haven't had ES "forget" my index since upgrading.

Some 1.x, and I don't intend to upgrade to 2.x, as it requires Kibana upgrade, and I don't want to deal with 150MB Node.js insanity Kibana 4 brings.

Curious if you had a chance to look at Fluentd (disclaimer: I'm one of the maintainers of Fluentd) If you did, I'm interested in your honest feedback.

Not yet, but I'll give it a shot. How's sharding/scalability?

We already have a transport pipeline; what's needed is a decently scalable, searchable, easy-to-use datastore for short/medium/long-term log analysis.

Right? An ES replacement in Go would be so great.

Why do you care of the language?

For one thing it means you wouldn't have to screw around with the peculiarities of the JVM.

Ridiculously easy deployment and lower memory footprint.

Is there already Lucene for Go? ;-)

http://www.blevesearch.com/

It has an very long way ahead to catch up with Lucene, but it's a very promising start.

Naïve question from a bystander: how much better is Lucene compared to bleve ?

/aside: I get rewriting for the n-th time some utility (say a db driver) in language du jour, but (possibly poorly) replicating things like Lucene never made sense. Lucene is a fairly solid tech by all accounts.

Reply on news.ycombinator.com