I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
As Elasticsearch is closed source, you can also choose to switch to Opensearch which will of course not get any of these changes. Though I would not be surprised to see this move mirrored on their side as it makes a lot of sense to do this. But it would end up being an independent implementation of the same concept.
As a long time Elasticsearch user, this stateless architecture makes a lot of sense to me. Especially for very large clusters. Basically, it vastly simplifies scaling and cluster operations. You can literally auto scale nodes both on the indexing and querying tiers. That's a big deal. You want it faster? Add more nodes. Likewise upgrades are a lot easier. Simply bring new nodes online and re-index to some new objects. Once it is done, take down the old ones. Also it simplifies testing. You can simply bring up a few test nodes and query your production data without having to worry about affecting production loads. Everything gets easier.
It will be interesting to see how they will bring this to market. This reads like they are starting the work on this, not like they are ready to deploy this. I guess this would be part of a future major release and they just had their previous one fairly recently.
Elasticsearch meets the above definition.
The permissiveness of the modification and enhancement is a different topic.
This seems right in theory, but when you actually look at the details it stops being true.
Specifically, Lucene today is best served through memory mapping the files and using crazy amounts of RAM.
My guess, Elastic just no longer cares about latency sensitive users. Even going from ES 5 to ES 7 there was a performance reduction for queries using any multi-word synonyms. It’s likely this trend is set to continue with remote storage.
I still think it’s the correct trade off for ES. Most of their clients use them for latency-insensitive analytics/log type stuff.
Elastic had to do this in face of competition from the likes of quickwit.io and snowflake search. AWS might have something similar up their sleeve, because I don't believe Athena/Trino can do super fast searches, yet.
So you wouldn't need to use S3 or some s3 emulator like other comments have suggested.
Give the ES cluster a disk it can write to that is guaranteed to be replicated (ie a RAID cluster) and thats it. The limits of object storage (no appending, no in place editing) tend to make those clusters cheaper so you could also run something like Ceph locally. It's probably going to do a better job of persisting data than ES also.
If this is the future of ES I don’t see a happy path for non-cloud customers…
Do you mean they "only support"?