gsteph22

Oh, and as far as scale-wise: we run on the "web scale", which means a dozen to hundreds or more of computers, each generating tons of logs. It's why we're built on Hadoop. It's the main complaint our customer research indicated people have with tools like Splunk.

Looks like I got cut off in the middle. *You can write Pig or Hive scripts to generate interesting analytics.

Thanks for providing this valuable perspective. I sort of group grep/Perl scripts/everything else together as manual processes. What I was getting at is the whole "roll your own" scripts is a royal pain in the ass.

For analytics, you're right, search is only part of the equation. That's why we make MapReduce easy to use on a cluster. You can write Pig or Hive scr

We also have templates for common data formats (and ways to roll your own) so you can turn unstructured log text into structured data, so that a histogram of hit counts per day-of-week is just a few lines of a script (or maybe even a search).

Thanks! Actually our software runs in the cloud or in your datacenter. If it's in the cloud, it can be encrypted in trasmission (or stored encrypted if you're willing to take the performance hit).

If it's sensitive data, I'd recommend just spinning up your own cluster and installing the tool.

We have tried them :) And so have several of our customers.

It's true that LogSearch is similar, but we focus on cloud, analytics, ease-of-use and scalability -- each of which we'ce heard Splunk lacks.

Flightcaster does some fantastic, innovative stuff in the statistical, distributed computing, and Clojure worlds.

Fully scalable, faceted, bolt-on search :)

We're good. We just want to be "more good". Bootstrapping cash.

I'm the guy who quit with OP. In fact, his lead engineer (on paper). There's an interesting dynamic when two people quit at the same time. Game theory time!

We do have a fair amount of leverage when it comes to negotiating severance pay, due to our knowledge of distributed architecture.

But yes, we're committed 150% to the startup. And submitted our YC application a few days ago :)

This is actually quite salient advice. There's so many blogs that just say, "OMG do it!". You need to be rational and have a plan.

Hey there! I'm one of the guys building the scalable, bolt-on search. Thanks for the mention :D

The fact that our government is moving towards Open Source to power not only the "Smart Grid", but the lifeblood of our country, is huge. It took some ballsy, persistent work to get this done.

Erlang? Thrift? Tasty!

It's a Secret!

Oh, how horrible -- self promotion :)

That's interesting -- we've seen a lot of the opposite, especially from the .20 release. I guess it all depends on what you need to do with it :)

It's my favorite article :)

I wrote a fairly detailed post about the Death of the Swiss-Army RDBMS on my blog, http://www.roadtofailure.com :)

The RDBMS has been used for way too long as a generic "bucket" for any sort of data storage and analytics. As a result, speed and scalability have suffered. With Social Media and the advent of "web-scale" data, engineers need to realize that the storage and analysis solutions they use must be driven by the type of data they're storing, and what they want to do with it.

As I stat, "The ACIDy, Transactional, RDBMS doesn’t scale, and it needs to be relegated to the proper dustbin before it does any more damage to engineers trying to write scalable software."