Elasticsearch from the Bottom Up (2013)

LMAO Ferret was sooo bad. It was written by a guy who effed off to Japan to study Jujitsu, and I remember my first startup job was porting all of its ints to size_t.

A footnote: For Java devs out there, the Lucene codebase is a joy to read [0]. Esp, the APIs. Highly recommend it just for the documentation which is top-notch. Michael McCandless, Lucene's committer-in-chief, sometimes blogs about its internals [1].

[0] https://github.com/apache/lucene-solr/tree/master/lucene

[1] http://blog.mikemccandless.com/2019/10/concurrent-query-exec...

Or, instead of porting Lucene, just take its main concepts such as analysis, tokenization, an in-memory trie or binary search tree, query parser, term, query and collector and implement them the way _you_ would and using whatever bit maps and other file formats _you_ see fit when serializing to disk. If you iterate enough times you'll realize that as you have grown in your capacity of understanding these concepts your code base has turned into quite an approximation of Lucene, with the same flaws and the same strengths.

If you feel Lucene is close to a global, or at least a very high local optima, then by now you know search.

Now the real fun begins, because now you get to implement your own search model as either a Lucene model or one that is supported by your own code.

There are ten or twelve really interesting problems to solve before you can call it a day. Before you're done, you'll start to see everything and I do mean everything as a search problem. Use Wikipedia as your tutor but prepare yourself to have your current world view become completely transformed, because what's a search problem, really?

What's a word? What's a phrase? What's their meaning? What's this word's meaning in the context of these other words? What's a paragraph? What's the meaning of this paragraph, in the context of these other paragraphs? What are some of the patterns we can perceive in the binary representation of our data? What are some of the patterns we can perceive in the vector space representation of our data? The answers to these questions and more lies in the search model and implementing one is the most fascinating thing there is, because no one, not even Google's top engineer, knows what the correct set of questions are.

Nicely written posts that helped me better understand what's going on, thanks for writing.

Vespa looks pretty good, at least in terms of performance and operation. I've been evaluating it myself. I'm less happy about everything else.

It's got a mishmash of odd APIs, lots of XML, several query languages, lots of weird little quirks. It doesn't feel modern. It's pretty clear that this is originally an in-house project, developed over many years by many people, where not as much effort has been spent on consistent/cohesive design or documentation.

One rough area is the approach to schemas and indexing. Rather than let you define a "clean" schema and put in your data and then have Vespa index it in all the ways it knows about, you're forced to essentially reshape your data into a format compatible with Vespa, which brings with it some severe restrictions. For example, Vespa will not index arbitrarily nested structured data. If you have something like {categories: [{id: 1}]}, Vespa will not index that. You have to flatten any array data to the top level. Nested maps and arrays are mostly not supported, although it's hard to tell from the documentation what is supported.

Vespa is also very obviously skewed toward ranking, not filtering. You can't search by exact string matching: You can't do something like "topic = 'news'". You only get case insensitive substring search. It's got lot of ranking functions but very little that's optimized for filtering.

Overall, I'm a bit surprised that Vespa's authors position it as an Elasticsearch competitor, because you certainly cannot just port an app that uses ES over to it.

To be sure, it's got lots of interesting features such as ML integration, and, again, performance and clustering design seems good. But it still feels very much like a niche product.

> JSON logs with a timestamp + a bunch of fields then search them

There is S3+Athena for this with AWS and Google can store/query JSON with BigQuery. The nice little UI doesn't come with it, but at least you don't have to spin up an Elastic cluster.

I mean, that product is called LogDNA.

But Elasticsearch has evolved into a whole bunch of things to meet everyone's needs. There's a way to do what you want and simply, but you have to find the simple path in the middle of the big product.

I think the question you should be asking is, does my use case require a tool as powerful as ES?

And only you can answer that fully.

I mean, the same reason that SQL isn't any simpler? It takes arbitrary data, and you can filter and aggregate it in arbitrary ways? People aren't just doing log data... it's just been heavily adopted for that purpose

I think the "simpler" version of ES+Kibana is probably a spreadsheet.

I built https://logflare.app for exactly this.

My team is using Loki + Grafana and we're pretty happy with it. It's pretty basic but it does what you expect it to do just fine.

We ditched Elastic as it was a super massive PITA to operate (& a resource hog at that). I'll admit I'm not an expert at ELK at all, but tbh I was absolutely surprised just how bad Elastic + Kibana was for our basic log uses when they tout it as one of their mainstays. Or we were just exceptionally stupid, who knows. In any case, the experience we had with it didn't motivate us to become ELK experts at all. Our pet peeves:

- The Kibana UI needlessly wastes tons screen for whitespace

- makes it hard to dig down into logs

- never seems to find exact string matches when we wanted it to and instead returns "helpful" fuzzy matches

- Kibana has no qualms sending requests to Elastic that will happily kill your node instead of applying sensible paging / query timeouts. I mean that's why I'm using Kibana and not writing my own elastic frontend...

Solr is quite a bit more straightforward than ElasticSearch, but still a bit complicated in my opinion.

ElasticSearch is really just clustered Lucene with some nice features wrapping it. You can probably get away with something that also wraps lucene. Though elasticsearch has a dominant position precisely because it is quite full featured and easy to run.

if the logs are only a few GB you can still write to files and fgrep.

Splunk is that product, but it’s only free up to 500MB/day of indexing