Hacker News Reader

Comment by nissimk | Hacker News Reader

Is there a lightweight solution for centralized logging with full text search and a nice UI? ELK seems like it requires some serious RAM and cores even if you are just talking about elasticsearch. Is there a similar solution that works well in just a couple of gigs of RAM for a slower message flow rate?

Is anybody here successfully using ELK in a low ram environment with low message flow?

View on HN

If you're open to commercial solutions, there are several SaaS choices that are quite cheap at low log volume. For instance, Scalyr (500MB/day for $19/month), LogEntries (free up to 167MB/day), or Papertrail (33MB/day for $7/month). All will provide centralized logging and full text search, and some form of dashboards and alerting.

Scalyr provides especially powerful features for log parsing and analysis, as well as integrating system and application metrics, and it's wicked fast -- most searches run in well under 1 second. (Disclosure: I am the founder of Scalyr.)

If you're interested, the respective web sites are easy to find, or drop me a line (my email address is in my profile).

For most teams, I definitely recommend getting started with a hosted elasticsearch option initially, and after validating a POC, only then consider a move to self-managed. Check out https://bonsai.io as a hosted option...

I've helped a bunch of financial and security companies implement and deploy elk as a critical infra component. Generally I'm seeing it used to complement Splunk, Alienvault and other log/network/security monitoring solutions.

Happy to help with any questions about deploying and managing ELK in production.

I haven't found ELK to be terribly resource intensive. We're currently loading ~20,000 events / min into an ELK stack that runs on a single m1.xlarge box, we do use redis as a logstash broker to deal with unusually bursty load, but it doesn't seem to have much issue keeping up with normal load.

Did you need a customized server configuration of Elasticsearch to enable this kind of throughput? And do you split your indexes up by day or some other criteria?

We're running a similar configuration and I'd like to know the limitations before we'd need to start using a clustered setup.

Nothing special on the ES config. Indexes are split up by day and we avoid indexing fields with very high cardinality.

Thanks for the info. Does avoiding indexing high cardinality fields keep down the memory usage of an open index?

I assume it would reduce the disk usage as well.

Memory usage and CPU time to maintain the index.

Please define low RAM.

I've got a t2.small server on AWS that's taking in a low/moderate logging from three servers (syslog x3, nginx x3, zero-low traffic redis) and it's usually in the 750MB of RAM with absolutely no notable CPU load.

Granted Kibana/ElasticSearch gets pretty much no traffic other than checking it once or twice a day, just to get a glance at 4xx/5xx errors.

So yeah, you just defined my current use-case down to the letter -- centralized logging, fancy interface for filtering/searching/visualizing said log data.

I am running a monolithic installation in a container (1 core, 4gb ram) that is being hosted by an intel NUC. I'm taking in about 100k records a day from 10-12 syslog sources and have no trouble with slow downs at all. (I do almost 0 processing of the incoming data, however.) I think I could lower the containers ram allocation to 2gb... maybe.

I haven't tried aggregating a years worth of data into a single search, but for reasonable log troubleshooting everything is going well.

Have you tried it yourself on your dataset? I've been experimenting with ElasticSearch on small datasets with a single node on my laptop and it doesn't seem to be hugely resource demanding.

Some people like using Grafana with InfluxDB backend and Fluentd for log collection.

Reply on news.ycombinator.com