Comment by chrissnell | Hacker News Reader

Comment by chrissnell | Hacker News Reader

Parent Full thread

Ding ding ding ding.

It's not the log storage that we care about, it's what we can do with those logs.

I've been through three logging vendors in the last two years (Loggly, LogEntries, now Sumo Logic) and I'm looking to move once again. The secret sauce is not the storage volume; our volume is relatively small: 4.3 GB/day. The secret sauce is the intelligent structuring of unstructured log data, analytics+alerting on the now-structured data, and live tail'ing of logs.

Sumo Logic does an okay job at the parsing/querying/alerting/analytics piece but live tailing is a brand-new feature for them and it has limitations. Sumo Logic's pricing is also frustrating--they don't apply volume discounting until you're pushing something like 500 GB of logs each month. Again, we need affordable analytics; storage is a small piece of the puzzle.

Honestly, our next move may be to an internal "HEK" stack: Heka, Elasticsearch, and Kibana. I've looked at the cost of doing this and it's actually more expensive when you consider what we'd spend to host a big ES cluster internally, but cost is not our #1 consideration, usability is.

This is exactly the sweet spot we're focusing on at Scalyr: quick, easy, and powerful parsing, analyzing, and alerting on unstructured logs, including realtime alerts and live tail. We have customers who are infinitely happier after migrating from Sumo Logic.

Congratulations to the LogDNA team on the launch. There are a lot of use cases in this field and it could use some more shaking up.

(Disclaimer: I'm the founder of Scalyr.)

I'm currently using Sumo Logic. I'm pretty happy with the features, but their pricing is pretty frustrating and limits what I can get out of it without multiplying the cost.

Scalyr looks interesting, how does it compare to Sumo Logic feature-wise? For reference, this video has made me switch to them: https://www.youtube.com/watch?v=w5yWfYDQz2Q

We have a pretty strong feature set for ingesting, parsing, and querying / analyzing log (and metric) data. Our key differentiators from Sumo are performance (real-time ingestion, 60-second alerting, extremely fast query execution), pricing, and simplicity / ease of use. We focus on engineering use cases as opposed to business intelligence, so we don't (yet) have all of the group-and-join functions that Sumo provides for creating BI reports.

Speed turns out to be surprisingly important. We've had a lot of customers report that after migrating, usage across the team goes way up, simply because it's much more satisfying to use a tool that gives quick results. See https://www.scalyr.com/case-studies/returnpath for a brief case study.

I'd be happy to discuss in more detail, but I don't want to hijack the thread - please drop me a line at steve@scalyr.com.

Have you looked at Splunk Cloud? Having used both it and the ELK stack, I can say that Splunk lets you easily do way more with your data than ELK does.

http://www.splunk.com/en_us/products/splunk-enterprise/featu...

http://www.splunk.com/en_us/products/splunk-cloud.html

But cost is the problem, isn't it?

The main reason we emphasized storage is because we built it with scaling that in mind, that's all. We've also built it with live tail in mind too. It's an okay differentiator since we believe we can handle the volume with our infrastructure. We also followed up with UI/UX which at the end of the day should make us a platform you can goto, get what you need and go back to work.

Give us a try, even if you don't think you'll use the GBs. We have alerting, field parsing and live tail. Analytics is the only piece we don't have today, but we're working towards it.

I understand. For me, though, scaling is just a small piece of the puzzle and not really a differentiator. To give you an idea of our size, we run about ~500 instances in a private cloud hosted on about 45 dedicated servers at Rackspace. We use OpenStack for stateful services and data storage and we use Kubernetes/Docker/CoreOS for 12FA/microservices. We have many dozens of apps and even with all of that, we're still only pushing about 4-5 GB of logs each day.

The number one most important problem we're trying to solve is the actual parsing of logs during ingestion. We need logs from commonly-used apps & frameworks to be automatically identified and parsed and we need an easy way to set up a custom parser for internal logs that doesn't require crazy hand-written regex. How does LogDNA handle parsing? What apps/frameworks are supported?

We automatically detect common formats using regex during ingestion. Many are based on logstash patterns and we have several of our own (including one that just tries to strip out timestamp + log level). If you use a common format, we can usually add it. If you have a custom format that your app defines, we don't have a way for you to add it yet but it'll likely be a regex as you said. (We can always help you create it if you don't want to).

Give me syslog target, and I'm all yours :)

Do you really need a large cluster?

We run our ELK stack (elastic search+logstash+kibana+3 days of logs) on a single EC2 ephemeral instance inside an auto-scaling group. If we lose our logs, oh well! Given that we lose this instance every two-ish years, it seems like a reasonable tradeoff.

Well, we use Elasticsearch extensively for our applications and have developed our internal best practices for running it. A single instance is not recommended for ES. If you're totally fine with losing logs, I guess it's okay, but logs are a critical part of production infra for us and so we would either run a larger cluster with data-only and master-only nodes for optimum reliability.

Amazon now offers Elasticsearch as a service; it was a great help for us quickly setting up ELK to test.

Yes, but it's limited to version 1.5.2. Given the backwards-incompatible changes in 2.X, that can be a deal-breaker for many people, sadly.

I have used/enjoyed scalyr in the past. Pretty good dashboarding, alerting, etc.

HEK sounds like: https://github.com/mozilla/MozDef since Mozilla makes Heka

Have you tried that? I wonder how Mozilla likes it

Reply on news.ycombinator.com