ES seems to be the most popular but only in its ELK avatar for devops dashboards.
What technologies are you people using for text or document or website search nowadays?
For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support.
Lucene is the core engine behind both of these solutions.
For fun, lets look at the large Enterprise acquisitions over the years:
* Verity - bought by Autonomy
* Fast - bought by Microsoft (Also known as the Enron of Norway...)
* Autonomy - bought by HP (Look at the backstory on this deal!)
* Endeca - bought by Oracle
* Vivisimo - bought by Oracle
* Google - GSA (now Google Cloud Search, hosted solution)
Next, follow the path of online acquisitions:
* IndexTank - bought by LinkedIn
* Swiftype - bought by Elastic
There's a number of interesting independent players still. Coveo plays in the Enterprise space, but it's a hard market. Algolia is doing great in the commodity online search space and seems to be growing well.
This is an area I think is open to more competition. Especially with AI/ML technologies available around Document Understanding - the Enterprise market is open for a good on-prem upstart to really take off.
Ping me offline if you have additional questions - spent almost 20 years in the space and ran a search company of my own.
That one was painful to live through, we got forced to migrate to Windows and everything went sideways. That was almost 10 years ago with quite a big cluster (tens of nodes).
Interesting, could you elaborate on why SOLR is dominant in that space over say Elasticsearch?
Also, relevancy in retail is often influenced by other factors that cannot easily be implemented in Elastic. TFIDF/BM25 search is available in both platforms, but you may also weigh in other factors such as relationships with the vendor, stock on hand, or other ML techniques that are more easily implemented in SOLR.
These days, I would say there's very little that either product does that the other product can't do though obviously there are lots of strengths and weaknesses on both sides.
Thankfully we are moving to Azure Cloud Search services, which just work, over the next year.
Good bye manure pile, hello compost.
Source: I have the capital V from the building in my house.
(I see now atambo has already noted this typo)
While I have not used Solr recently, it has evolved along with ES as a solid product with a solid community. Nothing against it; it's a solid choice and there are probably people offering to host that as well. Either way you are using Lucene. Using Lucene directly makes no sense unless you know what you are doing and have a need to do that. If you have to ask that means it's not for you.
If search is important to your use case and things like relevance, precision, and recall have real impact on your business, you should get some specialists involved and not reinvent the wheel or make all the rookie mistakes. Somebody like me basically ;-). However, that can be expensive and if search is not that critical, just sign up for one of the search as a plug in solution type products out there and don't bother running a lot of infrastructure. E.g. Elastic offers a thing called App Search and it probably covers most of the simple needs and is stupidly easy to get started with. There are several competing products probably that I can't vouch for.
You can always upgrade to something proper later. Things like mongo and postgres also have some limited capabilities here and you can get away with doing some simplistic stuff with sql even. However, there's a point where you hit a brick wall and there's something you need that is simply hard/impossible using that or where you end up reinventing a lot of stuff that things like Elasticsearch probably do better.
I think everyone doing Elasticsearch well has to bring it in-house eventually. AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive.
There's a 7-figure/yr Elastic Cloud customer I work with who is so tired of Elastic just randomly killing their clusters out of nowhere and having to spend basically triple to deal with it that they're bringing it all in house.
Once you grow larger than that though, the hosted service prices get astronomical compared to standing up a cluster, assuming you have someone who can admin it.
My point is that a ton of companies (more for cultural reasons than business requirements IME) are so freaked out by any downtime in any way that they’re going to pay for an engineer to just maintain these things themselves.
If you're just spinning up something's defaults and throwing data in it, that bill is going to due eventually and it's probably going to be ugly.
What has been poor about it in your experience?
The backup issue has been resolved I believe but the others...
On AWS I’d suggest elastic on ECS and you can use the leftover compute on the cluster to run other applications effectively for free.
If a configuration change (changing # of nodes, instance type, etc) goes wrong, your cluster indefinitely gets stuck in Processing due to a race condition. The only way to get unstuck is to file a ticket. The company I'm at doesn't pay for AWS support, so at one point we ended up completely tearing down our cluster and rebuilding a new one (via Terraform) after getting tired of waiting for the reps. (They advised us to cut off log flow to let the system get out of processing, which we did, but it didn't work because once it gets stuck in processing like that it's just completely stuck).
It's difficult to troubleshoot issues - you can get some logs via Cloudwatch, but they're hard to search through and I'm not entirely positive everything shows up there.
Amazon is always several releases behind Elasticsearch versions.
--
Elastic.co's offering looks much better, just by reading their excellent comparison article: https://www.elastic.co/blog/hosted-elasticsearch-services-ro...
(We haven't used Elastic.co but what they say makes sense and I imagine their service is much better)
--
Once you hit a big enough scale - for us, we're pushing about 2TB a day of logs (that's before accounting for replication of course), it doesn't make sense to stay on Amazon's hosted service.
I'm in the process of advocating for in-housing our Elasticsearch setup and just building on top of ec2. Elasticsearh seems like the perfect candidate for Kubernetes since rebalancing is automatic and the affinity rules are simple (every Elasticsearch instance needs its own node). Cluster autoscaling (i.e. node-level, not _horizontal_) just makes too much sense.
Unfortunately I haven't gotten the go-ahead to take it inhouse, but I've been gunning for the project for some time now, so I'm hopeful I'll get the opportunity.
--
BTW, totally unrelated but for anyone managing Elasticsearch, make sure you have your shard count tuned properly. When I came to this company, they had their data way oversharded with primary shards varying between hundreds of kb to a few gb; i.e. orders of magnitude difference. Switching to ~50 GB shards (done via simplifying the way we were indexing) massively improved performance.
Also i3 instances > anything with EBS.
[/ramble]
ES is indeed super solid if you know what you are doing. However, most users getting started with this are probably going to find out a few things the hard way though; which is why I recommend hosted solutions as it removes quite a bit of non trivial devops from the equation.
That is because Elastic Cloud is not a fully managed Elasticsearch. People often don't get that with Elastic Cloud you are still responsible for your ES cluster. That's one of the differentiators that e.g. Sematext has (disclaimer: founder).
> AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive.
Right. Have a look at https://sematext.com/logsene pricing. People say Sematext compares favorably to Logz.io and Elastic Cloud.
The per gb cost on the other ones start at 1.20 $/gb and goes till 2$/gb. While almost all hosted ELK solutions start off at 3$/gb.
Im asking because i would very much like to adopt an ELK based hosted solution..but I'm not able to justify paying double. Is it that running+resource costs for ELK are so high that the extra charge needs to happen ?
You do not need an agent. You can ship directly from syslog ( https://docs.datadoghq.com/integrations/rsyslog/?tab=datadog...)
I've never had clusters being killed randomly but I did have a few self inflicted issues with cluster instability due to flooding the cluster with too much data and not having suitable data retention policies in place. With the recently added index life cycle management (an x-pack feature), this is easier to manage these days.
If you are spending seven figures a year on elastic search, you clearly are not a beginning user and there's some cost savings that you might be able to realize by taking ownership of the problem of hosting it somewhere cheaper. For that, their recently open sourced kubernetes helm charts are worth a look. Those scripts take care of a lot of things and get you a self hosted version of Elastic Cloud.
Amazon hosted clusters are indeed a bit bare-bones (as is their support for these clusters) and I would also not recommend them; you get more value for money by using Elastic Cloud.
Also, the new trend in jobs is "Relevancy Engineering", which is less about just setting up search engines and more on actually tuning them. That's where also Machine Learning and other AI techniques come in (Learning to Rank, Named Entity Recognition, sentiment analysis, etc.). Which was recognized by rebranding of the conference from Lucene/Solr revolution to Activate last year.
See also Haystack conference which focuses very specifically on relevance regardless of the specific search engine: https://haystackconf.com/
Was there an opinion in there as well that you tried to convey?
This expertise is very on demand.
My company personally migrated from ElasticSearch to https://vespa.ai/ and could not be happier. Faster and way easier to maintain a cluster. The "Application Packages" feature present in Vespa opened many opportunities to improve our product( Curiously we use Lucene inside our custom application for a "Search map" functionality, for something like that https://www.lexisnexis.com/en-us/products/lexis-advance/sear... ) . I highly recommend it!
For me Vespa is a absolute game-change, in features and as someone said here, ES looks like it intentionally complicated to maintain. With nodes randomly getting unhealthy. Vespa is like Redis to me. I completely forgot about maintaining it and works great.
It makes a world of difference in our product, and I take every opportunity to evangelize it.
This may be of interest to you: https://sematext.com/opensee/report/project/trend?q=ElasticS...
Would you happen to know how Vespa compares to ES in terms of memory or CPU footprint? Have you done apples to apples comparison by any chance?
It's also readily apparent that it's an ancient system that's grown out of an in-house project, and that its design has accrued a lot of oddities over the years from lack of careful, co-ordinated design. It includes a bunch of esoteric features (like the "predicate" function) that have obviously grown out of Yahoo's own internal architecture.
One curiosity is Vespa's approach to schema and configuration changes. To make any kind of change, or indeed set up an index, you have to create that "application package" containing your schema and configuration in the form of files, and then use separate REST APIs to "upload", "prepare" and "activate" it. There's a CLI tool to help perform those steps, at least.
It's nice that they're more consistent and rigid about schema and config evolution than Elasticsearch. But it's not exactly operator-friendly, at least not for first-time users with no pre-existing operations based around Vespa.
The package design also makes it more cumbersome to perform programmatic updates for a schema. I once worked on a SaaS project where we indexed data in Elasticsearch — arbitrary documents where we didn't know the schema ahead of time, because we just accepted any JSON document posted by the client. With ES, we could just use its dynamic mapping support, which automatically creates field definitions when new fields arrive (using regex-based templates). Do you know how long a package update takes in Vespa, to add, say, a single field?
The Vespa documentation is also pretty terrible, in my opinion. They explain a lot of things, but it's confusingly written, uses a lot of homegrown terminology, and neglects to collect all the reference documentation in one place. For example, you can't find an overview of the entire API — fragments of it are just scattered across a dozen or so unrelated pages.
Lastly, Vespa is Java. One of the biggest challenges maintaining Elasticsearch is controlling its resource consumption. You have to give it a lot of RAM, and it's never clear how much it needs and what configuration settings and usage patterns affect its memory use. Tuning it is something of a dark art. I don't know exactly how Vespa is implemented (is it all pure Java?), but I'm worried that, being a JVM app, it has the same shortcomings.
In my own attempt to compare the two, I found the memory consumption of Vespa was easier to predict and understand (there are formulas for it in the documentation).
If you have dynamic fields like in your SaaS example I recommend using a single map field rather than let data not under your control drive changes to the set of fields.
> Do you know how long a package update takes in Vespa, to add, say, a single field?
A few seconds. However, rather than having operators do any of this manually, set up an automatic process which deploys on each change made to the repo (i.e do CD).
> all the reference documentation in one place
Edit: Documentation says: "Accessing attributes in maps and arrays of struct in ranking is not possible". So maps aren't really usable.
Regarding how long it takes to update a field, the application I described would have to do this programmatically. It would have to keep track of all known fields in some kind of registry, and then if a new unknown field came in, it would have to perform an "application package" deploy just for that field, using the REST API. (Unless there's a less cumbersome way to do it?)
Reference docs: That's nice, but that's just a bunch of links. Good reference documentation has tables of contents. Bonus points for runnable examples in multiple languages. For an example of good reference API documentation, look at Stripe's [1].
{
"location": {
"city": "Washington",
"state: "District of Columbia"
},
"friends": [
{"firstName: "Bill", "lastName": "Clinton"}
]
}
An application's search module could flatten the location key (e.g. "location_city", "location_state") for simple attributes, but the same is not possible for the array, since there can be arbitrary array elements. And you can't split it to an array of strings:
"friends_firstName_elems": ["Bill"]
"friends_lastName_elems": ["Clinton"]
This seems unnecessarily restrictive. A search engine should be able to index the data you already have, not force the application to contort its data to whatever shape the engine requires.
Is there no way around this?
I guess you don't see much demand because for a lot of use cases the basic setups are good enough.
I ended up trying to reinvent Solr for the client, realizing after about two days of trying to reinvent stemming and indexing, that this was stupid to do on someone else's time, and called the client to tell them that I'm moving to Solr, and I got the project done before-schedule as a result.
====
I think for 99% of usecases (involving search), Lucene/Solr/ES is perfectly fine. However, I do absolutely hate that some companies have decided to make it their primary database.
EDIT: I just want to make it clear, I think it's totally valid to try and reinvent Solr for fun, or if that's something you're paid specifically to do; nothing is perfect, and I am actually a big fan of the "if it works, break it and make it better!" mentality.
TLDR: Lucene < 7.5 won't merge segments larger than 5GB (default) unless they accumulate 50% deletions.
Delivering a conference talk [1] later this year about it.
[0]: https://www.eivindarvesen.com/blog/2018/09/16/elasticsearch-...
[1]: https://2019.javazone.no/program/3f7cd8a7-a9ea-4874-a7dd-531...
I don't think the GDPR regulatory agencies are operating at a technical level that they would make an argument that it was not a good enough deletion.
Finally I have to ask this part: assuming ES is not your primary database, how does this get around the GDPR issues? If someone wants their data erased you are supposed to erase it from wherever you store data, I suppose this means ES when it indexes a primary store and finds it has deleted data actually deletes it but if it is told to delete something it keeps it around?
If it is easy to delete specific users from the primary database, the deleted users will naturally disappear during the next ES reindex.
Edit: The old index is deleted at the file system level.
If the reindexing occurs daily or weekly, perhaps this will satisfy GDPR.
There are other good reason to not use ES as the primary data store. First, it isn’t entirely reliable. It’s good and I’ve never seen a corruption, but ES and Lucene’s history isn’t as a reliable database. Second, if you want to change how you index, it is a bit easier to do if the source data is outside of ES.
How is GDPR compliance of having data in Elastic Search influenced by it being primary vs. secondary storage?
Basically: you reindex ES periodically, so when a user is deleted from the primary, it will disappear from ES upon the next reindex. The old index is deleted at the file system level.
at what point would you be able to 'delete' data without being in violation of GDPR?
Though the EU has said it will consider intention etc. there's really no way of knowing for certain until when and if it's settled in a court case.
Fix it 'til it breaks is what I always say :D
Needless to say my 3D printer had a lot of down-time haha.
I inherited a legacy Mongo solution, and all the data is duplicated and indexed in ES, so I've always wondered why we're using both. Mongo has none of the SQL capabilities that would make my life easier, and the types of queries allowed by Mongo could be done with ES.
What are the negatives of ES alone?
The v7 upgrade to a new cluster protocol (zen2) has improved things but overall the system has a long history of losing or destroying data. It's better to have a primary OLTP system that's ACID and reliable while using ES as the secondary search source. You can also remove the _source field if you just need matches without the original content.
It's common to see pattern used with a relational database since, as you can see, Mongo doesn't buy you much else as another document-store.
Thing is, there's heavy demand for something more performant than Elasticsearch, so eventually the market will provide.
Meanwhile, Redis Enterprise is trying to grab some 'share with RediSearch, which has some severe caveats IMO that make it not a great fit for most.
https://github.com/tantivy-search/tantivy
That said - it's effectively Lucene rewritten in rust, so the main win is some performance gains. Lucene has spent a ton of time getting the details right, and it's unlikely we'll see an order of magnitude of innovation in that particular space. At the higher level for querying / query understanding it feels like there's still more technological room to grow vs the lower level details.
It is not exactly a port but yeah. tantivy is strongly inspired from Lucene.
> Lucene has spent a ton of time getting the details right, and it's unlikely we'll see an order of magnitude of innovation in that particular space.
Have you checked out the perf gain in Lucene 8.0 ? Block-WAND proved you wrong.
Tantivy is a cool project, but I have to say the part I love most about it is your blog posts on it. They're a great introduction for people who are unfamiliar with the underlying tech of search engines.
Thanks a lot! I am not a native speaker, and I often feel very bad at conveying engineering concepts. The positive feedback is actually very helpful :)
Disclaimer: I’m a previous employee but have no economic interests in this as it’s not a publicly traded company.
Vespa [1] would like to have a word with you.
1) Cloud search service - You are less likely to deal with setting up your own instance and concerns that go with it (sharding, etc) because most Cloud providers offer either ElasticSearch as a service or some form of turnkey deployment. You still have to do some low-level work like score manipulation but you don't deal with as much administration.
2) Search as a Service - https://en.wikipedia.org/wiki/Search_as_a_service. There are several companies that provide a Search SaaS offering. Typically they provide value adds above just running your ES service. Often they will provide web crawlers so you can just point them to your domain or they might provide other datasource integrations like pulling content from a database. You get access to Solr/ES functionality if you want it but you can get search running without going to that level if desired.
Either way a Lucene based stack is still in use.
ES is not supposed to be your source of truth.
https://discuss.elastic.co/t/elasticsearch-as-a-primary-data...
Any changes to the schema, such as switching fields between indexed/not indexed/stored/not stored, requires reimporting the data to populate those fields, data which you're not likely to have if it was your primary store.
1. https://www.elastic.co/guide/en/elasticsearch/reference/curr...
2. https://www.elastic.co/guide/en/elasticsearch/reference/curr...
First there's the Docker + Kubernetes architecture that ES lends itself to really well. Then (depending on your use-case) there are concerns like hot/warm architecture, node types, ETL/indexing processes. ES recently moved over to openJDK, so there's a couple intricacies there (i.e. JVM heap size)
Then, there's document/query structure. In no particular order:
- Do you have any parent/child relationships?
- Do you have stop-word lists developed?
- Can search templates help your queries?
- How will you interface with ES? It has REST APIs, but it's recommended to not expose ES directly to your applications.
- Some advanced querying possibilities like customizing tokenizers, normalizers, and a bit of internationalization.
- Oh, we haven't even discussed security yet.
- Also, ES isn't meant to be a primary data storage. This is more so a "cache", but not quite like Redis. So, you'll need a DB elsewhere most of the time.
All of this changes depending on if you're using it for SIEM, e-commerce, AI/ML, etc. Also, Elastic now provides their own SIEM solution, a pre-built search solution (AppSearch + Search UI), built-in security features. Check out the new ES 7.2 update; it's kinda nuts.
My current employers uses ES - we're on 6.8, planning to move to 7 in a few months. Judging by the other replies here I'd say we have a reasonably large cluster (150+ i3.2xlarge instances, billions of documents), so tuning the cluster is very relevant to us. Could you expand on how things have changed with the move to OpenJDK?
I've seen some claims online that, contrary to what Elastic recommends in their docs, a few machines with huge heaps (100+ gb) is the way to go, rather than many machines with 20gb heaps.
Usually the recommendation is less than 32GB - this link has some more discussion about it: From https://discuss.elastic.co/t/es-lucene-32gb-heap-myth-or-fac...
It seems whether it's better or worse depends on your data set . But I would love to see tests of different kinds of workloads with large or smaller heaps.
Also, you have plan ahead and over-allocate or deal with fixed indexes/datasets. You also have to religiously monitor the garbage collection and deduce what's going on with search & indexing performance. When the situation changes you need to scale your cluster and re-index everything, which is not a trivial thing at most companies. I've seen bad situations at companies where it takes days to re-index a cluster and they're dead in the water until it's done.
And that's just the operations side. You have to make sure your data is flat (because nesting creates subindexes for Lucene and kills your search performance), that you only define in your index template the fields that you want to be searchable (and binary blob the rest), etc.
Generalising a bit, Solr is more targetted at enterprise search and unstructured content search (e.g. bundled with most content management systems), and Elasticsearch is more targetted at data analytics and structured data search (e.g. with the ELK stack). Again simplifying a bit, Solr can be a bit more configurable and/or work better with the types of data that benefit from more configuration, and Elasticsearch can work better with the types of data that work more "out of the box".
I'd agree that there don't seem to be a huge number of openings for specialist search roles or a huge number of people specialising in search, but it is often part of another role and there are often people who have touched on search in their roles. That suggests that many people are just using it with largely default setups. Having said that, things like advanced relevancy tuning, if you need it, is a very much under appreciated skillset, and definitely needs someone with good experience or ability to learn.
We're at a point where Lucene and family are used for increasingly sophisticated use cases. The commodity end of the market used to be dominated by open source (Solr connected to Drupal for example)
Now for commodity sites there’s so many SaaS search products it doesn’t make as much sense to hook up Solr or ES to make your blog or university website or whatever searchable. A lot of basic search use cases are covered by products where you don’t want to have to hire a team to manage search.
But at the higher end apps with search, customization, especially of doing domain specific relevance at scale, is often a product differentiator (but often not so important or weird you should write your own engine). So this is where these systems thrive...
Also, SOLR speed is almost directly proportional to disk speed. If you index is on solid state drives with high iops, you'll be fine.
Backing up a large index is a little painful too.
Pros:
- Managed search engine
- Great API / Developer experience
Cons:
- Cloud only makes it hard for local development
- Expensive (I guess it depends on the usage)
Would love to hear your feedback :)
One area where this might be less true is that the full-text search in Postgres & MySQL have matured to the point where some basic applications might reasonably decide that it's not worth using a separate service.
For Rust, there's Toshi: https://github.com/toshi-search/Toshi which is built on top of Tantivy: https://github.com/tantivy-search/tantivy
For C++, there's Xapiland: https://github.com/Kronuz/Xapiand
For Go, there's Blast: https://github.com/mosuka/blast built on Bleve: https://github.com/blevesearch/bleve
We just rewrote our internal search API from a windows service indexer with lucene indices and a vb.net SOAP api in iis to a netcore service, hosted in k8s, that splits out ingest, analysis, storage and queries into separate domains, with the writes going to Azure Search Service*.
Our use case might be a bit weird -- this app is essentially an internal API that supports the search needs of our other teams and their own products for our own internal software. It probably has 30 million records across a few different indices. We made the decision to migrate from lucene because of the ease of clustering elasticsearch. We previously achieved availability by just running multiple copies of the standalone service and doing smart health checks at the load balancer level in case a lucene index got corrupted and needed to spend a day rebuilding, but that didn't scale well for rebuild times, and we have been consolidating all of our legacy tech onto netcore and kubernetes.
Raw lucene was an order of magnitude faster than azure search service, but that's probably more a function of being able to essentially query the indices directly in memory of the webservice, as opposed to a slightly underprovisioned search cluster with all the HTTP overhead. We're migrating it to our own elasticsearch cluster right now for performance, cost savings and cloud-agnosticism.
We are also looking at releasing this as a k8s deployable product. It's all k8s services and gRPC already...
EDIT: This seems to help with the ranking problem: https://github.com/postgrespro/rum
We've run into some pain points like trying to index very large shapes into a geospatial index, but have workarounds for basically everything now. We also had a problem where when AWS had the outage around autoscaling groups a few months ago, we lost 3/5 of our instances and had to reindex some data from backups. That was the worst thing that's happened.
I'm sure there would be better/faster/cheaper ways of doing what we do, but for what we get out of the box for the price, it's going to take a lot for us to move away from it for now.
The ability to not only full text search but to do it fast, to tune the lexical behaviour (lowercase, plurals, stemming etc.) and to top it all combine geo search pretty much left any other solution in the dust. I even considered some paid solutions.
I also considered postgres which looked strong but I felt it’s be harder to set up these features and that the full text would be weaker although geo might be stronger but my geo needs are simple.
ES was easy to set up to do this, taking about 2 hours of tuning. I used AWS so I didn’t have to figure out how to install it. I admit I had a mental model of ES from ELK-ing at work.
At some point when the site gets more traffic I’ll tune the search so that rather than nearest matching, I’ll score bit the distance and the words and order by perceived relevance. Ie weigh up both how close something is with how well the words match.
ES is a pretty amazing tech and it’s the easiest way to set up a decent quality free text search for your site.
For small projects (for some 1000s of documents), I'd probably go with Postgresql FTS if possible. Sphinx/Solr for anything with indices smaller than a couple 100GBs After that, ES seems reasonable & worth the overhead
EDIT: my biggest issue with ES is that it seems to be specifically engineered to sell you support. So get a managed version if you can.
why do you think you can't throw 1TB of data on postgresql?
The problem with PG FTS is that it doesn't have advanced search functionalities (fuzzy matching, faceting, term distance, highlighting results) and it lacks the modern relevance scoring systems so that'll be the limiting factor instead of speed.
Would love to hear your feedback.
We use Solr to power our main, end user-facing search after migrating from a custom Lucene solution some years ago.
To me it seems Lucene based tools are the best for the job if the main thing you care for is having text focused search with a huge potential for extensibility.
But there are a lot of use cases where you will never need anything more than the base capabilities of this technology (so you can be served by something simpler to use or maintain nowadays) and there are probably a lot of use cases where your search will be mainly driven by vector similarity (in which case you are working around the limitations of picking a technology with another focus).
As far as jobs go, I'm not sure how in demand specialists are. After a few years of working in the field I had a look to see if I could leverage my experience to get a remote position and came up with pretty much nothing.
I think that we definitely have the best integrated and most full featured solution for Sitecore customers.
It's not only about querying but also having a UI framework, built-in customizable indexing, analytics tracking and access to machine learning available in one package.
Source: Am product manager for Coveo for Sitecore.
I work at a state university (with the associated purchasing ... issues :)) and we had some staffing issues, which they were very accommodating of from initial quote to subscription.
If we end up running into issues as/if we expand our usage (we're essentially only using Solr for the mandatory bits) I'll keep Coveo in mind. :)
http://highscalability.com/blog/2013/1/28/duckduckgo-archite...
But for searching data that can grow and change over time, it's hard to justify using Lucene directly anymore. Azure Search (I believe built on Lucene) is an awesome (but relatively expensive) SaaS solution that is far easier to manage than Elasticsearch.
I built several search systems for classified listing sites, something like solr is a life saver once you get enough traffic. Is much easier to scalate than a sql database, and you can do much more things. The easiest example is a facet, for example you make a search on a car listing site, and you want to show how many cars from each brand you are matching, with sql you have to make another query, while in solr you can get those results in the same query. Now, add the model, the color, the gas type, the transmission, the place, etc. that actually grows easily to something unescalable, while with solr you can do easily.
We denormalize associated records into one Index. And any record that we need to find based on user-defined queries will go through ES since it's much simpler to metaprogram queries across denormalized data (no conditional joins).
But after you've gotten over that, you realize that this new tool can do lots more things than just text search. Time series metrics, BI, predictive ML, APM, etc. with relatively little work. With Solr, you could do those non-IR tasks, but it's going to feel much more awkward, IMO.
Say when you've got 10k-50k contact details (name, email, phone) and you want to provide a quick, autocomplete lookup. I've used basic SQL string matching for this, but it doesn't catch mis-spellings and the rest.
Running SOLR or ES is overkill for this. Is there a tool that fits this niche?
You want to be looking for tsvector related stuff.
I used it a few years ago to do full text search on a smallish website and it was great, I gather it has improved further since.
Completion response-time will be slower than Solr, Elasticsearch, Algolia, etc... but if you're already running Postgres, this may be the fastest to deliver for you.
With this small quantity of data, usually the app's running on a small VM. I'm wary of running anything Java, having had it require large amounts of RAM before.
That said, I haven't touched JVM stuff for 5+ years.
https://github.com/typesense/typesense
Would love to hear your feedback :)
At Mozilla I work one project with a search component (https://crash-stats.mozilla.org/), and it uses Elasticsearch.
Also solr, to although a lot less.
The reason to use version 1.7 -- is faster percolation (reverse search).
I love the "just works" functionality and portability.
At a previous job we had two existing options for search, Postgres GIN or the heavyweight ES cluster with Kafka. When I recommended grabbing Xapain for simple indexing (2-4k records, toss the index whenever we needed an update was OK) no one would bite.
Most of the time it has been for performance reasons. SOLR / Lucene have very poor performance characteristics, especially when needing to support custom sort ordering and heavy use of filters.
On other occasions it has been because you can’t easily extend search indexes to more advanced use cases, like similarity-based reverse item search, collaborative filtering, more advanced treatment of cold start issues / bias towards existing popular content / trending search.
A lot of small or medium sized companies naively figure they’ll just use SOLR etc to get something out of the box, or for side channel problems that are smaller scale.
But you come to regret it pretty fast because you end up needing one standardized way to build and deploy search indices and it has to support all the bells and whistles that SOLR can’t _and_ be faster than SOLR, for the big product use cases.
A lot of product companies now are beginning to use word-vector approaches with nearest neighbor libraries like ANNOY, as the first solution instead of the solution you eventually have to migrate to when you realize SOLR does not actually support your use case, not even as a means to get it up & running quickly.
Instead, you want to custom build the data retrieval system so it’s tailored to your use case.
One example from experience was needing to add hard filterable metadata to an in-house search index. We solved this by actually calculating bit masks that represented all the filtering criteria and having a frontend preprocessor that would first restrict to the filtered subset and then do TFIDF-based relevance sorting.
Creating the bit mask tooling ourselves (instead of relying on whatever baked-in method of scanning items and filtering that comes with out of the box search engine tools) allowed us complete control over the trade-offs, particularly managing document deletions and optimizing run time performance in certain ways that just weren’t available in out of the box tools, as well as being able to integrate any in-house code into the search engine as needed (since the whole system was in-house code).
You want to create data models that are highly application specific, and then route data into them. The mistaken approach of one-size-fits-all tools, especially in information retrieval, is to pre-define the supported behavior of the application, like a web service wrapping a search index, with baked-in assumptions about the trade-offs and only limited support to modify or configure the trade-offs under the hood.
The gravest mistake is thinking just because your use case seems to function OK with those assumptions now, that you can marry yourself to the underlying data model. Then in the future you’ll hit the point where you have to throw it away and create something custom, but it will be far more costly to do so and extremely hard to migrate gracefully and ensure integrations are working.