FoundationDB Fault Tolerance Demo Video | Hacker News Reader

FoundationDB Fault Tolerance Demo Video

blog.foundationdb.com

32 pointsnlavezzo15 comments

At 4:40 in the video we do a network partition - which seems to be relevant today in light of the aphyr / Jepsen post.

Also covered are single and multiple machine hard power failures, rejoining the cluster, and total cluster power loss and recovery, all in one take.

Happy to answer questions.

Question: Does FDB have any sort of "find" method in its API? As a key/value store I understand that I can set/get keys, but is there any way to find keys based on nested values?

I.e. fbd.find({someproperty: "123"})...

No, but you can utilize some data modeling techniques, such as the ones they mentioned on their website [0].

[0] http://foundationdb.com/documentation/beta1/data-modeling.ht...

A database is going to do a query like that in one of two ways: scan over (all of|a column family of) the data, or look in some kind of index on "someproperty".

In our approach this kind of functionality is the responsibility of a higher "layer" of the system above our key/value API. There are lots of ways of organizing and indexing data with different tradeoffs between search and update performance for different kinds of queries. We support pretty much all of them-- not by building any of them into the core, but by providing a general transaction facility that permits arbitrary indices to be updated.

For example: in a database like Mongo, queries like that one don't scale linearly even if there is an index, because you have to ask every shard to look in its local index. In FoundationDB, you would store the index in the database as keys like (someproperty_index, "123", _id) and an index read would only need to go to a single server.

It can be a hassle to create all the higher level functionality yourself, though! Over time, we and others will be offering more "opinionated" layers to save you time.

What a great demo! We're currently building a critical service on MongoDB and feel a bit leery about some of the issues others have expressed at scale. Would be interested in testing FoundationDB for our purpose in the near future.

Thanks! We thought since it was a hit in person we should share it a bit broader.

You mention scale - do know that what you're seeing in the video is probably the weakest FoundationDB cluster ever assembled :) Our in-house testing cluster is a 48 machine, 192 core setup, and it works great and scales almost linearly. You can see our performance numbers at www.foundationdb.com/performance (though those results are outdated now, the tests were on a cluster half the size and several versions ago).

If you'd like to chat about your particular use case, send me an email at nick at our domain.

Really interesting talk and I also just watched your "NYC Database Month" video.

What are the node costs in your licensing model ? Also any plans to release Flow as open source ?

Thanks. We haven't published pricing yet but we are aiming to make it in the ballpark of the hardware & operational costs of the machines required to run a particular size cluster. I.e. if you are running a FoundationDB cluster at 50% load then the licensing costs shouldn't be out of sync with the costs of owning and operating, or renting, the machines in the cluster.

It does look likely that we will open source Flow at some point, but it will take a good amount of work to do it properly and right now there is plenty of it to go round. Always good to hear interest for it though!

Great demo. Really curious about the nuts and bolts of the set up. What OS, and what is actually launching at startup? Again, very nice job.

I'd have to ask one of the guys that put it together for specifics in the morning, but the OS is Ubuntu Linux, and the applications auto launching are of course the FoundationDB server which then connects to the rest of the cluster and begins doing work, and also a client with a Firefox browser running the testing / demo app which increments the counter, changes the circles, and uses (I think) HTML 5 to render the images.

Kind of irrelevant, but just out of curiosity, but what are those little PCs and monitors? They look very interesting!

The boxes are by a company called Zotac. They're like $200 without drives and we put cheap consumer SSD's in them. The monitors are USB powered, and driven by USB - they are from a company called DoubleSight.

Funny story - given the amount of people at Disrupt that were not at all interested in databases, we had probably just as many people ask us detailed questions about "those cool little computers" as we did our database :)

Ah, I see. Demos are hard but you did a great job. Very well done.

Thank you, sir. We'll have some even cooler demo videos coming in the future!

Amazing demo. I'd love for aphyr to take a closer look...

Reply on news.ycombinator.com