FoundationDB community highlights, two weeks in | Hacker News Reader

FoundationDB community highlights, two weeks in

foundationdb.org

137 pointswwilson49 comments

To add to the list in TFA, Rust bindings are here:

https://github.com/bluejekyll/foundationdb-rs

You beat me to it!

I think we're getting close to a 0.2 release. We have enough working for the class scheduling tutorial, so seems like a good next point release soon.

I’m curious how they plan to do continuous integration. The Apple hardware-only clause of the macOS SDK license agreement really crimps the options for CI, especially for open source projects with limited budget. Travis is a godsend, but there are startup latency and capacity issues there even in the paid tier. Apple has made major contributions to open source, as evidenced by FoundationDB here, LLVM, WebKit, and other projects. But they no longer make servers, and the fallback — Mac mini — is on life support, which puts projects and service providers who need to treat macOS as a first-class target in a real bind. Even something limited like the IE Edge and Visual Studio images Microsoft provides would be a huge boost.

>I’m curious how they plan to do continuous integration. The Apple hardware-only clause of the macOS SDK license agreement really crimps the options for CI, especially for open source projects with limited budget.

FoundationDB is a distributed key value store. These almost always run on servers. macOS is not a server OS. Is there any reason to build this on macOS, over, say, FreeBSD.

Many developers want to install it for local development - presumably this is a particular popular sentiment _at Apple_. There is a FreeBSD port already, however [1].

[1]: https://www.freshports.org/databases/foundationdb/

Sadly, as I discovered today, the FreeBSD port is incomplete.

First up, it doesn't create the fdb.cluster file. I reverse engineered that logic out of the Mac OS packaging script, which wasn't a big deal. Once I created the file I was able to start it up and connect to it with fdbcli.

Second, the Python bindings do not work on FreeBSD. Trying to set the API version (the first step in calling the API) gives an error specifically saying they're not supported on FreeBSD.

Still though, for something that was released two weeks ago to be already ported to FreeBSD is pretty impressive!

There has been a more complete pull request [1],[2] back to the main project repository which solves some of these problems, and should ultimately result in the (FreeBSD) port being fairly straightforward.

[1]: https://github.com/apple/foundationdb/pull/309

[2]: https://forums.foundationdb.org/t/freebsd-support-for-founda...

>Many developers want to install it for local development

Why? Just use the in tree Dockerfile (which didn't work due to linking errors with libstdc++.a missing definitions for basic_istream::ignore last I checked).

Try release-5.1 branch. See https://github.com/apple/foundationdb/pull/263.

lol to webkit. It was a fork of khtml.

It is, but they improved and maintain it.

Is there a precedent for Apple enforcing this clause on providers in the CI space? I vaguely remember some cease-and-disist letters to Hackintosh vendors but I'd imagine that has to do with brand protection.

> they no longer make servers, and the fallback -- Mac Mini -- is on life support

Well, there's also the Mac Pro (companies even make rack mounts that fit 2 machines in a 4U). But I wholeheartedly agree about the CI issues (having been hampered by them in a previous project). That doesn't really seem to be an issue for FoundationDB though?

The only license you need to accept to use this software is Apache 2, so I’m not sure where you’re getting the idea that there’s an Apple hardware restriction.

The question is not about where FoundationDB can run in general. Sorry if that was unclear.

The question is about (public?) continuous integration testing for changes to FDB against macOS specifically, eg for pull-requests (for example, to allow contributors to make changes targeting other platforms without needing to own a mac for local tests).

Ah, sorry, that’s a good question.

The good news is that FDB has a fairly opaque platform abstraction, and deterministic simulation tests for everything above it, so the vast majority of continuous testing in terms of CPU cycles can be run on Linux.

The only thing that really needs to run on a Mac are tests for changes to the build or packaging system, or to the MacOS implementation of the platform layer. My sense is those changes are pretty rare.

I don’t know whether the FDB team has an official recommendation for MacOS CI. I’d recommend posting a question on the forums: http://forums.foundationdb.org

what's wrong with Travis? https://travis-ci.org/bluejekyll/foundationdb-rs/jobs/375032...

I already set that up, it installs the binary for FoundationDB then runs the tests for our Rust client on mac. No reason why they couldn't do the same for FoundationDB, but I bet they have something in house.

The Swift project uses an array of Mac Pros: https://ci.swift.org/computer/

I'd like to see the ANSI SQL layer open-sourced, and some benchmarks.

If you need SQL now, you might consider CockroachDB:

https://www.cockroachlabs.com/

It is on-the-wire compatible with PostgreSQL clients.

Except where it isn't: https://github.com/cockroachdb/cockroachdb-python/pull/14#is...

I apologize if I was being unclear.

It is wire-compatible with PostgreSQL clients, meaning that if you have a PostgreSQL client library for your favorite programming language, you can use that to connect to CockroachDB.

The database features and SQL dialect itself are not 100% compatible with PostgreSQL, and likely never will be.

Also, even if all the SQL you do want to use is supported, it is likely a sub-optimal idea to just port some SQL over to the CockroachDB platform without additional investigation. Some issues, like interleaving values from different tables, should be examined to ensure good performance on CockroachDB.

And even setting aside issues like that, the access patterns you'd use for a multi-region distributed database are going to be a bit different than a high-availability PostgreSQL cluster. Data locality and all that.

The wide availability of client drivers just means you can get started quickly on popular platforms.

What would be the point of benchmarks? Benchmarks say pretty much nothing about distributed databases.

But you do want to know how such database behaves in general and during failures with some load and some data, say 100M records and 3 nodes. What kind of latency it has when nodes are added, replaced, moved etc. Including 90th percentile latency. How long does it take to move 1 TB worth of records, what if this process was interrupted, can it resume later or not. How does it deal with HDDs and bad sectors, especially wrt to performance, since disks like to retry bad sectors for a few seconds, completely blocking all I/O, requiring special handling. Or is it only suitable for SSDs.

Last time I looked, the SQL layer was really slow. I know they invested a ton of effort to improving it, but I don't think they ever really cracked the problem and I think you're likely better off with CockroachDB, as mentioned in another reply.

I've built a toy SQL database before that actually did a good job with query planning and used existing highly optimized storage engines (lucene or rocksdb).

It was slow AF, because the actual query execution has a ton of method dispatches, etc to e.g. check index conditions or assemble rows and it never quite gets fully JIT'd by the JVM.

Every reasonable SQL engine i'm aware of compiles queries to lower-level code. A cursory inspection of their code indicates they don't do this, so its not surprising its slow.

https://github.com/jaytaylor/sql-layer Here's the old layer :)

It's even been ported and packaged for FreeBSD: https://www.freshports.org/databases/foundationdb/

I've also created a package, as well as module support, for FoundationDB on NixOS:

  https://hydra.nixos.org/job/nixpkgs/trunk/foundationdb51.x86_64-linux

Hopefully it will be in a good place by the time NixOS 18.09 is out.

Hoping someone makes a Swift interface, given this is coming from Apple, one would hope there would be one.

It's a race between a Swift native client and Swift-Python interop :)

I am very curious on what Apple uses as the server OS in its datacenters. Do they use OSX or do they use Linux / FreeBSD?

A company of Apple size likely uses multiple OS', I wouldn't be surprised if they even run Windows servers somewhere.

Judging from their "DevOps" position requirements, their stack is similar to many others out there Chef, Ansible, Docker, Kubernetes, Java, Python (with accent on Python so maybe Salt Stack), Ruby, Go, Git, C++.

Given their extensive Hadoop infrastructure, likely CentOS or Debian flavor. Some job positions list "RHEL or derivative a plus".

> Ansible,... Python (with accent on Python so maybe Salt Stack)

I'm not sure it makes a lot of sense to use ansible and salt stack (both automation tolls written in python).

They strike me more as alternatives than being complimentary.

Apple is a big company. Different teams/divisions within the company might plausibly use both.

> I wouldn't be surprised if they even run Windows servers somewhere.

I would be a little, because they seem to proud to do it. At least their public facing infrastructure seems very well selected and uniform.

Probably a silly question, but does someone running it in production already? I mean not Apple, but someone from community.

As the article states, Snowflake Data [1] and Wavefront [2] have been using it. SkuVault [3] is another company.

https://www.snowflake.net/how-foundationdb-powers-snowflake-...

https://www.wavefront.com/wavefront-foundationdb-open-source...

https://abdullin.com/sku-vault/foundationdb-layers/

There are probably several other companies from the original releases that haven't posted anything publicly, but it'll be awhile before we get full deployments from the latest open-source versions.

Is there any apache Spark foundationDB integration, like drivers?

You might be interested in TiDB's TiSpark which is all built on top of TiKV (equivalent to FoundationDB).

https://github.com/pingcap/tispark

Another highlight: FoundationDB is in nixpkgs unstable!

Could somebody clarify the license? I was excited this was Open Source, but then some people say you are only allowed to run it on MacOS? How can that be?

Also, how does FoundationDB compare (in general, and in performance) to:

- CockroachDB (competitor) https://www.cockroachlabs.com/blog/2-dot-0-perf-strides/

- Redis (competitor) https://redis.io/topics/benchmarks

- GUN (ours) https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2...

- Fauna (competitor) https://fauna.com/enterprise#Scalability

- VoltDB (competitor) https://www.voltdb.com/blog/2017/10/02/comparing-fast-data-p...

- RocksDB (competitor) https://github.com/facebook/rocksdb/wiki/performance-benchma...

Is the the big boom in FoundationDB just that it is "now Open Source", or that Apple vetted it (why didn't they keep it closed source?), or is there something distinctly unique about FoundationDB that it surpasses a bunch of other tried and true systems?

I think one major distinction is that Foundation DB can be used to build most of the databases you mentioned above because of its architecture.

I'm pretty sure that is true the other way around though:

Back in 2011 I built a couple DBs with Redis, and I've definitely seen people swapping other databases (MySQL, MongoDB, etc.) storage engine for RocksDB and others. I also know the storage engine we wrote (Radix Storage Engine) could be used to build LevelDB and Cassandra on top. Even Cockroach makes for a pretty solid storage engine.

Could you give an example of why you couldn't use VoltDB to get FoundationDB like properties?

It is possible to write code to add a compatibility layer on top of any DB to make it work like any other DB. Foundation DB's architecture specifically accounts for this. It was designed from the ground up to allow building different interfaces/layers.

We can write compilers for any language to compile it to Python bytecode. That does not make python VM and .NET CLR the same thing. CLR clearly has a design goal of allowing multiple languages to run.

> Could somebody clarify the license? I was excited this was Open Source, but then some people say you are only allowed to run it on MacOS? How can that be?

there's nothing to clarify. there are people spewing nutty weirdness, and then there's https://github.com/apple/foundationdb/blob/master/LICENSE

if that somehow isn't convincing, they distribute linux binaries.

Thank you! This is legit, glad to see it is Apache2.

EDIT: Why on earth are people downvoting this comment, is Apache2 hated by most HNers?? It is one of the best licenses out there.

EDIT 2: RE: @sigstoat comment:

I read through the whole article and even checked out and read through JosephG's (ShareJS author) FDB repo before commenting, and had even read the other comments (which is where I saw the MacOS thing). Just to clear the air (thanks @sigstoat for suggesting an answer).

I followed FDB back when it was proprietary too, which is part of my curiosity if proprietary -> OSS receives a bigger reaction than companies like CockroachDB who have always been OSS. It has taken CockroachDB years to get to 13K stars, compare to FDB's 7K+ in just two weeks, I'm really interested in how/why behind these mechanics.

> Why on earth are people downvoting this comment

all of your comments appear to be getting downvotes, probably because they look suspiciously as though you haven't even spent 45 seconds on the website, despite being in the database space yourself.

> Could somebody clarify the license?

The license is Apache 2.0.

https://tldrlegal.com/license/apache-license-2.0-(apache-2.0...

> I was excited this was Open Source, but then some people say you are only allowed to run it on MacOS?

https://www.foundationdb.org/download/

There are binaries available for macOS, Ubuntu, RHEL and Windows. You're allowed to run it literally anywhere you're able to get it to run. Instructions for compiling from source can be found in the Github readme.

> ...but then some people say you are only allowed to run it on MacOS? How can that be?

I believe they were positing that CI would be difficult, since Apple limits macOS to running on Apple hardware.

Reply on news.ycombinator.com