What are the downsides for implementing something like this rather than going for Elastic Search or SOLR?
Is it possible to have production-quality search using just postgresql?
Is it possible to have production-quality search using just postgresql?
The quality of search is a combination of both precision, ordering by relevance, and recall, finding relevant documents in a corpus. PG FTS is primarily a recall engine since it does not have BM25, TF/IDF, etc built-in. Out of the box, it is like an easier to use regex rather than a full search engine. If you have a dataset with small N or relatively unique records then recall-based algorithms will still give good precision simply because there aren't that many results for a given query. However datasets with larger N or lots of similar records, precision becomes harder and you start needing BM25, machine learning, etc.
It is possible to incorporate other relevance signals (eg recency, popularity) to improve precision in PG but depending on the specific signal, it can be more difficult than ES or Solr. Things like auto complete, supporting multiple languages, spell check, scaling are also harder to implement in PG at the same quality and performance as ES/Solr.
Syncing the database and search server is a real problem so I usually recommend to stick with PG if you have a recall-based search problem. But if precision is important or one of those other features I mentioned is important than I recommend ES/Solr and just deal with the syncing problem because overall it will be easier.
The downside: ranking can be slow when your keywords match a large number of documents (have a look at github.com/postgrespro/rum for a solution), and faceting is more difficult to implement than in ElasticSearch
ElasticSearch doesn't have ACID or very good OLTP manners. More importantly, it still continues to have data loss issues so it's not a good fit as a primary datastore.
Read the jepsen tests (old but show core problems that aren't fixed):
https://aphyr.com/posts/317-call-me-maybe-elasticsearch
https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-...
Or just search for ES data loss:
https://news.ycombinator.com/item?id=9475620
https://stackoverflow.com/questions/29841348/how-reliable-is...
And if that's not enough, here's the official page listing several "ongoing" fixes for major durability issues:
https://www.elastic.co/guide/en/elasticsearch/resiliency/cur...
I've never built something that did not need a relational database to model the domain, and search is such a common UX element that it's taken for granted in everything you use.
I can't even think of something that could use search that's not relational. Can you?
Elasticsearch stores data in a JSON document store style format. So any situation where you need O(1) lookup for all data related to a particular entity you can have nested structures within JSON that facilitates this. In a RDBMS it is O(n) since you have to do a costly join for each embedded structure.
360 customer views are good example of this.
If you want to save JSON documents, you can pick Postgres [1] or MySQL and still have a RDBMS for free additionally.
Elasticsearch does that too, but its access certainly isn't O(1) - it has a complexity of O(log n), just like Postgres or MySQL would have, because all of them use tree-like index data structures. (Please note that ElasticSearch is not meant as a data storage layer)
Now, if you just want to have the JSON, you're done. O(log n) for ElasticSearch, Postgres or MySQL. That's it.
But normally, you want to do something with that data. You have a user and want all the products. Or you have a book and want all the authors, because you have a real app to write.
Most data in our world is relational: A student can partitipate in courses, a store sells products, and so on. Queriyng your database using JOINs will make use of indexes in each of the tables that you have previously set up.
These indexes will prevent the theoretical worst-case (the "Full Table Scan", where your complexity is O(n) for the table). The query planner will decide which index is better to use for the optimal response time.
You can, however, choose to not leverage this feature of RDBMS. What do you do now, if you need all the courses of a student? Do you get all the JSONs and compare their IDs? Do you parse the JSON keys in your own app? But having your app iterate over all the JSON keys already has O(n) complxity. So you end up recreating tree-like indexes anyway somewhere in your stack. But all of that is already provided for you, with decades of debugging and performance optimization - for free.
To sum up: Use Postgres JSONB type if you only need to write JSON documents. You might later want to query the nested JSON data structure and join it. I highly recommend "Mastering PostgreSQL in Application Development" [2]
[1] https://www.postgresql.org/docs/current/static/datatype-json... [2] https://masteringpostgresql.com/
The comparison was between ElasticSearch and a relational store. PostgreSQL's JSON type is not relational. And if you are fetching any entity by a primary key then it should be O(1). The point was that document stores offer you the ability to nest data structures whilst relational forces you to join. And so if that is your query/data pattern then a document store can be orders of magnitude faster than a purely relational store e.g. 360 Customer View.
I've got a handful of situations where nested structures are making sense, but it doesn't seem to be the norm (at least in the data worlds I work in).
Primary key fetching will always be O(log n) due to how indexes are implemented. Access time complexity will never be independent of the item count in your data store, relational or not.
> PostgreSQL's JSON type is not relational.
It is. In Postgres, you can join nested JSON structures' keys with normal, relational tables: https://dba.stackexchange.com/questions/83932/postgresql-joi...
My point is: Postgres offers you a document store and gives you the option to join, if you feel like it.
More importantly, ES is not a document store. It is a search system that can be used as a similarity index between all kinds of data, most commonly json docs but can be anything like images or audio waveforms.
ES can return none of the source data or not even store it in the first place. In addition, it lacks ACID and transactions and generally has an overall poor reputation for data integrity.
If you just want a json blob, you can already use a relational database column for it and get the key/value performance semantics, while still retaining all the reliability and manageability properties.
Yeah, so perhaps you don't have much experience from outside the relational model. Makes perfect sense to see a single truth then.
> I can't even think of something that could use search that's not relational. Can you?
Not sure how wide a blanket you're throwing by "not relational", but here are a couple examples that might fit: Google's search engine, most filesystems and their files, hierarchical and flat data structures in general, twitter.
It depends on your search use case. If you are doing fairly simple searches such as trying to find, say, usernames, you can certainly have production grade search using Postgres.
However, if you have to handle searches that are not as targeted, Postgres will not be enough. One example is searching with synonyms - if you want a record containing "automobile" to match queries containing the term "car", ElasticSearch will serve you better.
Postgres is a relational data store and ElasticSearch is an information retrieval system; each focuses on their intended use case. The fact that Postgres supplies some elementary search capabilities does not mean that Postgres can well serve a bona fide information retrieval user case.
If you have a solution like Citus available to scale your DB horizontally, it might be enough forever though.
Simplicity of having it all in 1 system is great, however the search quality itself is usually average and configurability is limited. A real search engine will give you much better relevance rankings, facets, fuzzy search, and other fancy things you can do with a general "similarity index" like recommendations.