Searching a data breach with ElasticSearch | Hacker News Reader

Searching a data breach with ElasticSearch

adamfallon.com

41 pointsAJRF12 comments

I had heard no shortage of warnings against using Lucene/elasticsearch to host my own basic search. But it was incredibly easy to implement and about as painless as running an apache server. For a self hosted project I can recommend it. Is it Algolia? No. Did I need Algolia? Also no. But it has been running for 5 years with barely any need to touch it once it was set up. Happy days for a hobby project, or a project with on-premises requirements.

Agree, I think after using it in professional context I can say it scares me to think about running it on a heavily trafficed site, but it is amazing and easy to set up for hobby stuff and even small side projects. I've replaced it recently with Postgres FTS though (https://www.postgresql.org/docs/current/textsearch.html)

These were my major pain points with PG FTS.

Lack of ability to search indexes by relevance OR date

the highlighting function randomly returning no values despite a hit

lack of ability to define relevance by word positions - searching for "red dog" returns "the sky is red, i love my dog" and "red dog" as the same score. RUM indexes helped, but did not fix #1

Almost all security issues with ES stem from their idea to keep authorization as separate, paid product (X-pack). On other other hand MongoDB had similar issues since they wanted to their product to be easy to setup and use, maybe for people scarred of pg_hba.conf.

Actually wrong at several levels. First, the free version includes all the security features. Xpack was always free (as in beer) and easy to enable when that still was a separate thing (it no longer is). After they changed the license for the whole of elasticsearch to be similar, xpack is no longer a separate thing. You just get the whole product. For free. There are some paid features in there but they are mostly related to high value stuff related to e.g. machine learning.

The "security issues" mostly stem from people intentionally running it without a firewall completely unprotected on the public internet. And then they put important data in there. Simple solution: don't do that, it's stupid and negligent and it's 100% your fault if data leaks like that.

You wouldn't run a database on the internet either. But if you must run it on the public internet, just put nginx in front of it with basic auth and https. Problem solved. Not that hard.

Alternatively, use the hosted version which doesn't allow you to do that at all and is nice and easy to get started with.

To be fair, "X-pack" was a thing many years ago. But those days are gone and have been for a while.

You get authorization in the free offering and that's been the case for the least two major versions.

The things you need to pay for are IMHO not hobby project stuff, like integration with LDAP/AD.

You are of course correct that this used to be the case, and that to some extend this sentiment prevails.

But I feel Elastic (the company) deserves credit for acknowledging the issue and addressing it.

That used to be the case, buy the Opensearch fork forced them to include it in their free offering

What is algolia?

a search SaaS

It's also the search API used by the search box at the bottom of every HN page

I opted for Solr over ES due to the craziness of ES licensing, and other fiascos they have had over the years. Solr seems more "open" and in line with what I believe.

Sure, Solr is rough around the edges but it does virtually the same thing.

You can always use OpenSearch, which is equal to ES 7.x (since it was forked from there, before they went rough with the license)

Reply on news.ycombinator.com