Building With Workers KV, a Fast Distributed Key-Value Store | Hacker News Reader

Building With Workers KV, a Fast Distributed Key-Value Store

blog.cloudflare.com

46 pointsjgrahamc11 comments

I love it and want to use it, but two announcements of a new cloud hosted distributed DB and no mention of how consistency works or how conflicts are handled worries me a bit.

Maybe one of the cloudflare people that read HN can shed some light on this? What happens when I write to the same key simultaneously from multiple workers and/or multiple edge nodes?

> What happens when I write to the same key simultaneously from multiple workers and/or multiple edge nodes?

It's eventually-consistent. Eventually, one of the writes "wins".

If you do a read/modify/write in the context of processing a single request from a Worker, we'll automatically apply optimistic concurrency so that the write fails if another write has occurred on the same key concurrently.

But otherwise, yes, the ability to handle transactions is limited. For now, Workers KV can be used for a variety of use cases but is not a full replacement for a database.

Workers KV is only the first of multiple storage-related projects we're working on. Stay tuned. :)

Hi skrebbel, storage is my project at Cloudflare.

Last write wins, for some definition of last (https://en.wikipedia.org/wiki/Eventual_consistency).

As we see how initial developers make use of the storage, we are planning to add additional synchronization and consistency functionality. For example the ETags model using If-None-Match and If-Match would be appealing to some.

They write it in the description: eventual consistency with global consistency within 10 sec.

That's rather vague definition. And is also wrong. They obviously can't guarantee global consistency within 10 sec, as that would require consensus which is impossible to guarantee in bounded time and they probably don't even use any consensus algorithm at all, as that would be pointless for a CDN.

Yes, in the presences of a network partition, propagation can take arbitrarily long. But, that would be classified as an outage. Knowing the typical non-outage delay is useful.

> Reading values from Workers KV is designed to have the same reliability as reading static files, making it much less likely to become unavailable than a traditional database. It’s designed to have the same performance as reading a file cached within our network, close to your users, giving it the speed of serving a static file as well.

I wonder how this kind of speed is achieved with an API that has to go over the network? Even if the round trip is short such as between two AWS services, there's always at least some latency.

Reading cached files goes over the network too. CDN servers share the cache load and have just 1 or a few copies per datacenter that are served by every machine there.

It's likely that this KV store is built on top of the existing cache storage layer, which would also explain the eventual consistency and high-reads with low-writes.

Fantastic development. Cloudflare is making some great progress recently with workers.

Listing keys in a namespace and/or TTLs would be nice with KV.

Going a bit off-topic, I can't access this CloudFlare blog using a Tor browser (using the "standard" security level) without a CAPTCHA.

Didn't we discuss just last week an article[0] by CloudFlare where they said they'd solved this problem and Tor users wouldn't need to solve CAPTCHAs for CloudFlare-fronted services anymore?

What happened to that? I've actually seen way more CloudFlare CAPTCHAs through Tor since that article came out.

[0] https://news.ycombinator.com/item?id=18031649

So... A hosted Consul?

Reply on news.ycombinator.com