Open-sourcing Facebook's internal branch of Thrift | Hacker News Reader

Open-sourcing Facebook's internal branch of Thrift

code.facebook.com

66 pointsjamesgpearce14 comments

This is very cool but forking your open source project like this strikes me as very poor open source behavior. Now there's Thrift and FBThrift. It would have been much better to have integrated improvements back into the open source Thrift project incrementally.

After thrift became an Apache project, internal Facebook employees found it harder to iterate using external tools than our internal github repos. Many of the changes depended on things that weren’t open source, or only recently became open source in folly. We do hope to merge as many changes as possible back upstream, but felt it was more important to get the code open source as quickly as possible. In addition, fbthrift is a dependency for several other projects Facebook wants to open source in the near future.

Wouldn't it have made more sense to contribute the changes first, rather than creating a fork of your own project?

This is just wasteful behavior. Why would anyone try to work with you to build community around this code? Why should anyone trust that you won't just abandon this fork's community and then come out with another full rewrite in a few years time from your silo?

It doesn't sound to me like their intention is to build community around this code. In fact they explicitly state their intention to merge everything they can back to Apache Thrift, and the fact that they've been tracking upstream closely.

Working exclusively through open source communities like the ASF can slow you down sometimes, so if you need to iterate quickly on something sometimes the upstream review or release process isn't appropriate - even though it might require more work later. They're still open-sourcing their technology, and intend to move it back to the original project. What more can you really ask them to do?

> What more can you really ask them to do?

Nothing, of course. But it would certainly be wise for anyone to relying on their code to reconsider: if you try to participate in the community, you may simply be wasting your time.

I find the part where they fork their own project darkly amusing.

But the fact that they've open sourced that fork, and are working to get changes merged upstream is great news. It's far better behavior than many other companies: Oracle, IBM and Sun all come to mind.

Google use an internal fork of Protocol Buffers as well. They also have a Protobufs based RPC stack that they've never released.

Personally I commend Facebook for their recent open source contributions. Sure, they suffer from bunker syndrome, but that seems to be par for the course for these sorts of companies. Just look at Android for another example of code-drop Buckaroo.

It sounds like a significant amount of the friction may be coming from the Apache Foundation's infrastructure and/or bureaucracy – or am I reading that wrong? It is hard to understand why Thrift isn't hosted publicly on GitHub – especially when Facebook is using GitHub Enterprise internally – unless the choice is political.

> Many of the changes depended on things that weren’t open source, or only recently became open source in folly.

Is "folly" some sort of jargon I'm unfamiliar with here, or does this literally mean that some things were foolishly open sourced recently due to a lapses of good sense?

https://github.com/facebook/folly

I'm slightly disappointed that it wasn't meant as a literal folly.

I hope one of those is Haystack. I've been dying to play around with it. It was originally supposed to be open sourced but they changed their mind.

I used to be all about Thrift. I first started using it back in 2008 and thought it was awesome for a long time. While it clearly works for Facebook (and many others), I've always had little problems with it. Connections silently fail (next request times out, following requests work again) if you don't make a call in long enough with some of the bindings. Generated servers sometimes leak memory. The type system is nice but creates all kinds of headaches when trying to reconcile it with something idiomatic in different languages. When many different services need to talk to each other it requires a lot of connections, each of which I came to view as a liability. Thrift's semantics are actually underspecified (http://twitter.github.io/scrooge/Symantics.html) so Twitter, who uses Thrift, came out with Scrooge, which is mostly compatible but not always.

My lesson was that Thrift is great if you have people who can look after it all the time. Thrift connections were never something you could forget about for a little while if you had a small team, though.

We've been moving away from Thrift towards a message queue-based system (using RabbitMQ) that lets us do everything Thrift does and more, and works every time. It used to be so nice to start a new service by writing the IDL and getting everyone to agree on a clean, well-bounded interface. Facebook forking their own project is kind of a last straw for my faith in Thrift, though. Oh well.

I wonder why they didn't publish the changes as one big monolithic commit on top of the Apache Git repo instead of obliterating all the history.

30k qps? That seems pretty low even on a per core basis.

Reply on news.ycombinator.com