Apple Open-Sources FoundationDB Record Layer | Hacker News Reader

Apple Open-Sources FoundationDB Record Layer

github.com

243 pointstiagocorrea32 comments

Announcement: https://www.foundationdb.org/blog/announcing-record-layer/

HN Announcement comments: https://news.ycombinator.com/item?id=18906341

I didn't expect this to be written in Java. Do they talk about this decision anywhere? I didn't catch anything in the announcement.

They acquired the company that made FoundationDB:

https://techcrunch.com/2015/03/24/apple-acquires-durable-dat...

This makes sense. Thank you for clarifying!

Now this is the kind of Embrace, Extend, Extinguish I like. I wish more companies bought technologies just to open source them.

The 4 year gap in the middle is unfortunate though, there could've been much more momentum by now had Apple kept it open all this time.

> ...4 year gap...

I don't think it was open in the first place, was it?

It was iirc

FoundationDB was not open source before it was acquired by Apple. They had a free version you could download and use, though.

There was no gap.

Also, it’s just as possible that open sourcing it earlier could have just bogged down the decision making because of alignment issues and caused it to fail.

Open sourcing is great and the more companies like Apple do it the better, but assuming that earlier is always better doesn’t seem justified.

FDB itself is written in Flow, which is 98% C++ with a few extra keywords thrown on top that gets compiled to actual C++. This gives them actor model concurrency in C++ that is completely deterministic by generating very ugly code that uses callbacks instead of threads or coroutines.

The reason for Java being used here, if I had to guess, is they have experience running Java in production and the Java bindings are the most mature (in my opinion). Writing the record layer in C++ would probably be overkill. The reason the Java bindings are the most mature is probably because they needed to be to write this :)

Wow people really have no imagination when it comes to naming languages.

Could you please provide a link to Flow?

https://github.com/apple/foundationdb/tree/master/flow

> Writing record layer in C++ would be overkill

Mind to elaborate on why C++ would be overkill? I understand the layered architecture decouples data model from underlying storage so it can be implemented in whatever language. To me, Flow and not having GC makes C++ a good choice.

It would be a fine choice, (and the document layer is) but the most performance sensitive parts of the entire system when you take it as a whole are already written in Flow. Writing it in C++ entirely would take a lot longer.

Apple's server-side WebObjects stuff is all written in Java; and IIRC, the original Control Panel for OS X was in Java using the ObjC/Cocoa bridge stuff.

It still makes me sad that Apple dropped the Java Cocoa bindings. That could have been a really great thing for cross platform development (probably the reason).

I so miss EOF[1] and really wish it had made the transition to OS X.

1) https://en.wikipedia.org/wiki/Enterprise_Objects_Framework

I believe a lot of their backend stuff is in Java.

This was a little disappointing to me; the paper[0] pitches as a design principle it’s efficiency and embeddability but it’s not clear to me how one could achieve this except in another JVM-based language. This is in contrast to the core fdb client library, which is ~easy to embed all over as it’s in C.

So it seems like services in languages like Go, Rust, python, node, etc are essentially forced to dream up a bespoke wire protocol and revert to the client-server approach. Am I missing something?

[0] https://www.foundationdb.org/files/record-layer-paper.pdf

I think the idea is that this layer could be exposed via a networked API. Though now there would be another layer of services to manage with all the associated, deployment, security, operability issues.

It wasn't directly stated as the reason for this choice, but being a heavy user of Cassandra and migrating away from it likely played a factor. It may have been that they had already developed a lot of supporting functions around Cassandra in Java and naturally led to the development of Record Layer in same.

Note that there's also a separate MongoDB compatible layer: https://github.com/FoundationDB/fdb-document-layer

And a copy of the original SQL layer: https://github.com/jaytaylor/sql-layer

When would we use this instead of a normal relational database? At which scale?

I'm surprised to see Apple using Github; particularly now that Microsoft owns it. It is where the developers are though. And it certainly beats a hidden download on opensource.apple.com.

The opensource.apple.com site is really more for software that they snapshot and release (open source, not open process). They've had other projects for a long time developed in the open, and used to run https://www.macosforge.org to host them.

Some of those were acquired (either as a core developer in the case of CUPS, a codebase in the case of WebKit) but I'm not sure of other examples of open sourcing an acquired technology like this.

I’d say Microsoft owning it makes it more attractive rather than less.

Less risk of it being acquired by some unknown company in the future or collapsing through mismanagement.

I'd go one step further and say that to Apple, Microsoft is "the devil you know". Apple uses Azure for some of its services, for example.

This is not exactly a new thing: https://github.com/apple/swift

There's also: https://github.com/apple/foundationdb

Cool. Is anyone working on a graph layer?

TL;DR to save y'all looking for it: Released under Apache 2.0 License, see file ACKNOWLEDGEMENTS for licenses of included 3rd party code (some BSD style, some more ASL 2.0)

Reply on news.ycombinator.com