Warp: Lightweight Multi-Key Transactions for Key-Value Stores | Hacker News Reader

Warp: Lightweight Multi-Key Transactions for Key-Value Stores

muratbuffalo.blogspot.com

Warp is quite interesting, and seemed really promising and ahead of its time. Unfortunately, Hyperdex (which is the key-value store that implements it) has not been maintained for many years [1], and the website is dead.

While the technology is impressive, the author seems to have lost interest, and moved on to working on something called Consus [2], which has also been abandoned (no activity 2018).

Hyperdex never became popular, and a problem all along was that the author — a very talented developer, from what I can tell — seemed more invested in his projects from the perspective of academic research (he developed Hyperdex at Cornell, I believe) than in delivering a practical, living open source project. He tried to form a company around Hyperdex (the transactional Warp add-on was commercial) even though nobody seemed to be using it; and he was the sole developer. I actually submitted a PR at one point to fix a build problem, but the author was completely unresponsive; you can't really do open source that way.

I think the Warp code was actually open sourced when the author realized they had failed to commercialize it, but I'm not sure; it's been a long time.

[1] https://github.com/rescrv/HyperDex/issues/233

[2] https://github.com/rescrv/Consus

Replying to myself that the author posted some information about their plans for Consus [1] back in 2016, with some commentary about how some people on HN were speculating about what exactly happened. Consus is also abandoned now, of course.

[1] https://hack.systems/2016/12/29/why-consus/

I mean, why can't you? If it's actually openly licensed you're free to fork it and fix the bits you want and use what you need.

The ability to allow contributors to conttibute code is a bonus of it being open source but even if it doesn't accept or merge pull requests, it's just as well imo.

Forks are expensive, socially, organizationally, and so on; especially for a project that has no mindshare to begin win. Forking a project just to fix a build problem is productive.

Too many projects just dump the code onto Github with no attempt to create a community around it. Community participation is ultimately what makes or breaks an open source project. I'm not saying one has an obligation to the world to service pull requests, but if you release something and make zero effort to engage, then don't be surprised if your project dies.

In this case, the codebase is also sufficiently complex/arcane that it's hard for anyone to just pick it up and evolve it meaningfully without any aid from the original author. Without knowing all the details of Hyperdex's internals, I do think that it might be a worthwhile thing for someone with sufficient time and interest to resurrect it, though.

You must know how difficult and complicated projects such as databases are.

You need to be an invested person to casually read the sourcecode of postgres or random database project.

I have some ideas on HTTPS://GitHub.com/samsquire/ideas4 of how I would cause programming to be easier to pick up and run with a piece of mature software. There is inevitably A LOT to learn to onboard yourself with a large codebase, it takes months or years to get familiar at work so it would take the same or longer for open source.

Can't edit, the "why can't you" was a reply to "you can't really do open source like that"

I implemented multiversion concurrency control[1]. I am interested how transactions are implemented without it. You need snapshot isolation too so you cannot read dirty writes and avoid phantom reads.

PolarDb sets commit timestamp to null and asynchronously changes the commit timestamp after a commit but has a CTS ring buffer for transactions that are still active. The reading transactions has to check this to see if the transaction is part way through committing.

The getting the behaviour of all at once or not at all is an interesting problem to have.

I use a database timestamp to solve this problem which is the latest transaction to have committed.

I think postgres does something clever with minimum and maximum timestamps on each tuple.

[1]: HTTPS://GitHub.com/samsquire/multiversiom-concurrency-control

This system doesn’t support interactive transactions does it? In that the whole read / write set needs to be known up front before a transaction can start being processed? I know that systems like FoundationDB and Calvin/Fauna work similarly and get incredible performance because sequencing is so much easier / lock free. I think those two systems couldn’t be adapted for interactive transactions really (without client side retries) but maybe warp could be which is cool!

FoundationDB does interactive transactions.

Last I checked, the entire transaction is built in memory (and famously has a five-second time limit), so while you can make multiple round trips, it's not quite as useful as with other databases.

The 5s timeout is annoying sometimes, but it does really support interactive transactions in the exact same sense that PostGres and MySQL do.

Nice: ACID guarantees; 75% of the throughput of the non-transactional key-value store it builds upon; linear scaling; conflict retries. Worth trying IMHO.

What's wrong with making your transaction a paxos(multi-paxos, raft,...) consensus value?

Nothing and e.g. Permazen does that:

https://github.com/permazen/permazen

Same name as the new Rust Terminal

Here's a nitpick: It's key/value store, not key-value store.

A key-value store would only store "key values."

I don't know of any rule of English grammar that would lead to this interpretation. If you do, you should immediately write to the maintainers of these websites:

https://redis.com/nosql/key-value-databases/

https://www.mongodb.com/databases/key-value-database

https://aws.amazon.com/nosql/key-value/

https://etcd.io/docs/v3.4/learning/why/

https://riak.com/products/riak-kv/

https://rocksdb.org/

https://github.com/google/leveldb

http://www.memcached.org/

The Chicago manual of style tells us that one should hyphenate compound nouns where both nouns are of equal importance; common examples include "philosopher-king" and "nurse-practitioner".

Besides, the slash is arguably more often used to mean "or" (think "he/she") than "and".

Haha. Key-value is meant (IMO) to show the connection between the key and the value. You're right, it's not a "key value" store, where "key" is an adjective modifying "value" to show that is only stores the most important values. The "-", again IMO, shows that there is a mapping between each key and each value. But geez, I think we knew what was meant :-)

I love how somebody modded INFORMATION down.

And here we were just making fun of Reddit for this kind of infantile behavior.

Because it's bike shedding. The post is about some pretty interesting database internals and a comment nitpicking about the semantics of language, which have no relevance to the article content, isn't constructive.

Reply on news.ycombinator.com