An incomplete guide to Facebook thrift | Hacker News Reader

An incomplete guide to Facebook thrift

avabodh.com

54 pointsrajeevk20 comments

We use Thrift a lot at my company, not just for internal communication, but for our external facing API.

Larger companies like Evernote (https://dev.evernote.com/doc/reference/) have shown that Thrift can be operated successfully at an enormous scale.

Facebook also re-forked re-open-sourced Apache Thrift (originally a Facebook project) as fbthrift: https://github.com/facebook/fbthrift

The unfortunate part of public Thrift APIs is that you must either provide the IDL and hope your consumers are smart enough to work with Thrift, or you have to take the Evernote route and provide SDKs for all the languages you care to support. Admittedly, making SDKs for languages is a lot easier when codegen takes care of 80% of the work.

Most options for an API offer you the same choice, no? If you're offering XML either you offer a wsdl and hope consumers are smart enough, or you provide SDKs for all the languages. If you're offering JSON you have to provide SDKs for all the languages, since there's not even a standard IDL.

> If you're offering JSON you have to provide SDKs for all the languages, since there's not even a standard IDL.

I disagree. It's best to, but if someone wants to hack together something with python/requests it's not hard to use the returned dictionary. Even with Java, we'll frequently do some requests to get representative payloads, generate models, clean them up and we're on our way. Or you could just use Map<String, Object>.

Does anyone know of a data serialization framework that will do the usual structs and arbitrary size matrices? I'd love to use something like thrift for serializing matrices or other larger data over a wire or onto disk but last time I looked thrift, protocol buffers and avro were all struct oriented and there wasn't a good way to parcel up a matrix or large vector. Currently I've just been using hdf5 as the default way to store reasonably large data matrices, cubes, vectors, etc.

What's wrong with using a repeated float to represent a vector/matrix, perhaps with a repeated int for indices if you are using a sparse representation?

Yes, I could take some time and write it myself. Take a look at hdf5[1] and the features they have around meta information, data chunking, page level compression for efficient reading of sub areas of data spaces. It is a non-trivial amount of functionality but allows some pretty amazing performance, for example see what pytables [2] has built on top of it. Now if they had just made it easier to attach it to a wire as well as a file.

[1] http://www.hdfgroup.org/HDF5/ [2] http://www.pytables.org/moin

Lisp (Clojure, ...) reader and pr would do this, no?

I found out the other day that Packetbeat supports decoding Thrift structs:

http://packetbeat.com/docs/configuration.html#thrift-configu...

Which makes debugging Thrift-based architectures significantly easier.

This is honestly a better guide to Thrift than we have inside Facebook, though I wish that said more.

Thanks you.. What more you expected in this?

Nice introduction. I selected Thrift as middleware for a business project for the company I'm working on one year ago, so I'm pretty familiar with it. It has some issues but for the most part it is reliable, fast, and extremely flexible. Problem with Thrift is mainly the lack of documentation and examples, so the more, the better.

I have similar experiences and generally like it. We had to fix one ways in Java at one point (now fixed properly) and not all languages have equal support. We wanted SSL support on a .NET service but nothing official there last I checked. FB updated and standardized SSL on C++ but left out other languages by and large.

There is a native version of Thrift for Java: https://github.com/facebook/swift

And a Scala one (https://github.com/twitter/scrooge, idomatic Scala generated models/parsers) when paired with Finagle/Thrift is pretty great.

"Facebook re-open sourced the thrift..." cool, let me get right on that before it's re-re-closed sourced.

Good to see this.

I was sold on Thrift but I think the popularity of REST made people take some weird decisions.

Internal services talking to each other using REST seemed like a bit too much, a place, where I though services like Thrift would have worked so well.

I think you mean HTTP+JSON. REST as a philosophy while applied mostly to HTTP isn't really tied to it.

We personally decided to stick with HTTP (even though Thrift would be "superior") since tools are numerous and simple. It's hard to beat curl and a browser. We also found that services end up used in ways that are hard to predict, especially in bash-written tools and the like. Thrift is a barrier to entry.

Additionally, you could do HTTP+Thrift. It's possible, it works, but you lose some of the overhead benefits of Thrift at that point (IMO). In my experience, fulfilling a Thrift request with a Thrift server has tighter latencies at much lower CPU usage for similar request loads.

It's a tough tradeoff between performance flexibility. Relying on HTTP + JSON is very nice when you have lots of teams, codebases in several languages, and want to preserve the ability to talk quickly to any service over HTTP.

I like working HTTP but certainly tools like thrift have there. LinkedIn is all about http://rest.li/ for our services though I've been working with DropWizard on a side project and really love it.

this is an interested introduction.

Reply on news.ycombinator.com