Stateless – Evolving the architecture of Elasticsearch to simplify deployment
elastic.co
elastic.co
A cloud offering that decouples storage from compute makes this a lot easier and becomes more of a no brainer for 90% of the use cases.
If you're an outlier with TBs or PBs of search data this probably you can keep using on prem if you want. Though I don't really immediately grant that it's worse for your usecase, especially when object storage is insanely cheap and supposing they provide a compat layer for S3 you can get away with everything from R2, S3, Ceph, Minio, Backblaze, etc. This is very much a case you should benchmark/analyze as a proper engineer.
I've managed many TB ES clusters. This would have reduced my cost and time to manage it so I could focus on other features that would have benefitted my users.
In the case of both services, they make the processes easier but you still have time and expense considerations to them, if only just waiting time and not active work on your part. But it also means you have a lot more monitoring concerns you have to implement and then account for.
Plus your choice in storage IOPS greatly effects query time so you also have to consider that and high IOPS storage is expensive.
It's a lot of things for someone who just wants a workable search for their product to have to know.
The new architecture they're proposing seems to allow for much more seamless scaling as your data grows without a bunch of manual intervention, monitoring and infra knowledge to make sure things don't fall over on you.
This, absolutely. I'll add that I once tried to look at the Elasti.co offer for an ELK solution that could handle the logs we are currently managing internally (on EC2 managed by us with some automation), and you basically had to specify all the details of the topology to get a quote. It was basically a layer on top of AWS Cost Calculator with their margin baked in.
No way you can tell them "I send on average X bytes/day, I want a quick answer for Y days and a slower answer for Z days and a global retention of XY days" and get a "you will pay aprox NNN dollars/month in the HA config, MMM in the non-HA config"
Interesting that ES might also end up with this kind of an offering.
I imagine that, especially after the OpenSearch debacle, they're keen to make sure they don't lose even more marketshare so they're paying particular attention to the way the wind blows and wanted to do this PoC so they can at least compete with people already in the stateless space like Quickwit.
This is music to my ears! Analytical databases are all so damn complicated to manage with a dozen of different stateful node types
As search engine that operates at any kind of scale needs to skip through very large files to evaluate a query. You need very low-latency, high-bandwidth access to disk. A search engine instance that accesses files on a local SSD is an order of magnitude faster than one that puts files on EBS.
They make some mention of local caching, but the devil is in the details here. Does all data get copied to local cache? What is the performance here?
I know Presto isn't focused on search, but Athena (AWS branded Presto) can do some really fast queries over S3, the issue is coldstart time on the compute, for a similar solution focused on search maybe you keep the compute always warm and work from there.
If you are doing multiple dependent loads, e.g. loading an index that tells you which part of the data to load which tells you which other related table to look into (e.g. a complex join)... that would be bad.
I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
If this is the future of ES I don’t see a happy path for non-cloud customers…
Do you mean they "only support"?
As Elasticsearch is closed source, you can also choose to switch to Opensearch which will of course not get any of these changes. Though I would not be surprised to see this move mirrored on their side as it makes a lot of sense to do this. But it would end up being an independent implementation of the same concept.
As a long time Elasticsearch user, this stateless architecture makes a lot of sense to me. Especially for very large clusters. Basically, it vastly simplifies scaling and cluster operations. You can literally auto scale nodes both on the indexing and querying tiers. That's a big deal. You want it faster? Add more nodes. Likewise upgrades are a lot easier. Simply bring new nodes online and re-index to some new objects. Once it is done, take down the old ones. Also it simplifies testing. You can simply bring up a few test nodes and query your production data without having to worry about affecting production loads. Everything gets easier.
It will be interesting to see how they will bring this to market. This reads like they are starting the work on this, not like they are ready to deploy this. I guess this would be part of a future major release and they just had their previous one fairly recently.
Elasticsearch meets the above definition.
The permissiveness of the modification and enhancement is a different topic.
This seems right in theory, but when you actually look at the details it stops being true.
Specifically, Lucene today is best served through memory mapping the files and using crazy amounts of RAM.
My guess, Elastic just no longer cares about latency sensitive users. Even going from ES 5 to ES 7 there was a performance reduction for queries using any multi-word synonyms. It’s likely this trend is set to continue with remote storage.
I still think it’s the correct trade off for ES. Most of their clients use them for latency-insensitive analytics/log type stuff.
Elastic had to do this in face of competition from the likes of quickwit.io and snowflake search. AWS might have something similar up their sleeve, because I don't believe Athena/Trino can do super fast searches, yet.
So you wouldn't need to use S3 or some s3 emulator like other comments have suggested.
Give the ES cluster a disk it can write to that is guaranteed to be replicated (ie a RAID cluster) and thats it. The limits of object storage (no appending, no in place editing) tend to make those clusters cheaper so you could also run something like Ceph locally. It's probably going to do a better job of persisting data than ES also.