FB's New Real-time Messaging System: HBase to Store 135+B Messages a Month | Hacker News Reader

FB's New Real-time Messaging System: HBase to Store 135+B Messages a Month

highscalability.com

68 pointskapilkaisare18 comments

Maybe it's just me being naive, but facebook has got to be one of the most awesome places that there is to work.

I swear that every couple of months I hear about a situation that basically comes down to this:

Boss: "Hello, Ryan, we would like to do $impossible_thing""

Ryan: "Huh, that sounds kindof impossible. But I guess we could get a team of the smartest people in our industry together and build an entirely new suite of tools to make $impossible_thing into $totally_normal_thing..."

Boss: "Yeah, do it!"

6 months later

Ryan and boss (to internet): "Hey, guys, check out this thing we made. Here, you can have it for free. Have a nice day!"

This makes me being too dumb to work there burn even more.

Likewise. In my experience with them, the interview questions are really specific and targeted, thus making it unlikely for everyone to know the precise answer. Unfortunately, they are indeed looking for an exact answer.

So just because we couldn't recite an exact response with compilable code doesn't mean we're dumb. We just aren't experts in that particular topic. Hang in there!

>This makes me being too dumb to work there burn even more.

I looked at your profile and saw this:

>a failure of a hacker who doesn't have the energy to do much of anything anymore

Failure of a hacker? Pardon my language, but I fucking doubt it. Prove me wrong and tell me why you're a failure, please.

I find myself falling into this ridiculous trap all the time. It's easy to do too... I mean, holy shit, I'm posting on hacker news, a place where multi-millionaires posting about their wildly successful startups is the norm.

I'll find myself thinking that I'm a failure because I don't work at facebook, and I'll probably never be rubbing shoulders with people at the googleplex. I mean...my god, what a total worthless waste of space I am that I'm not a member of the top %0.001 of programmers in the US? Because my office isn't in some uber-cool San Francisco loft? Because cisco isn't employing a team of engineers to try and build switches that can handle the amounts of traffic that my data-center crushing websites produce? Because I'm not routinely completely changing the game in application development?

That's ridiculous. I really really really highly doubt that you're a "failure of a hacker", the fact that you post here pretty much proves that you're not.

Huh, that sounds kindof impossible. But I guess we could get a team of the smartest people in our industry together and build an entirely new suite of tools to make $impossible_thing into $totally_normal_thing...

HBase has been around for a while, and is part of the Apache Hadoop project. You don't need Zuckerberg signing your checks to work on cool projects along with really smart people.

I'm also talking about cassandra.

Out of curiosity, am I wrong about this? Cassandra was developed by facebook, wasn't it?

(I was asking because this was downvoted. I thought maybe I was mistaken about cassandra)

Yes, it was.

But their decision to choose HBase over Cassandra should be commended and to build on top of a new tool or create custom tools frequently, make them really cool. For me, they look cool by just sharing this. Not many big companies do this.

Very true, but on the other hand, those who do work on cool projects might want their checks signed by Zuckerberg. (Not me, but that's a different matter.)

...those who do work on cool projects might want their checks signed by Zuckerberg.

I can't understand why that would matter, and frankly I find the notion very depressing.

I don't find it depressing that people want to make a living from the kind of stuff they like to do, even when it's open source. If I work on database technology and some giant company with huge amounts of data wants to use it, that's an interesting challenge. That said, I would very much prefer to get my pay checks signed by someone whose vision I support, or ideally sign them myself. So, personally, I would not work for Facebook.

You don't need Zuckerberg signing your checks to work on cool projects along with really smart people.

Being able to deploy to 500MM people has got to be pretty cool, though.

I am currently working on the HBase project at Facebook. It's extremely fun! We try to sync up with the OS branch on a weekly basis. Additionally, FB actually has an Open Source team whose primary job is to find out how we can use current open source solutions or remove company-confidential material and open source more of our infrastructure. It gives you real pride to know that your company completely supports your desire to help the software developer community.

Always remember that you are literally doing what people like me daydream about.

That's awesome :)

idk if you really associate Facebook with quality engineering quite yet. Facebook is often broken with rough edges, odd engineering decisions all over, and quirks galore.

It's very interesting that they didn't have the need to compromise on consistency and went with a fully consistent system, HBase.

I suppose fb caches the recent messages in memory, and use HBASE as archive storage most of the time.

It depends on the developer's favor to pick up a system. If it's me, I will pickup cassandra. The consistency model is not so hard to work with.

heh. I attended Jonathan Gray's talk at Hadoop World this year (which was pretty good!) and he talked a lot about Facebook's infrastructure. Gray has been a primary committer on the HBase project for a while (since inception?) and I remember people somewhat teasing that HBase was kind of the pet project of the NoSQL world. I've had fun hacking on it off and on myself. Gray amusingly said that HBase was being prepped for production in a huge way, but he couldn't give any details.

Well, I guess we know now! Definitely looking forward to all the future developments of HBase. There are a lot of interesting contribs being made.

Reply on news.ycombinator.com