Hacker News Reader

Comment by blackaspen | Hacker News Reader

Twitter's Github: https://github.com/twitter

Apache Aurora, Bootstrap, Storm, Heron, Pants, ThriftMux, DistributedLog, Apache Cotton, Iago, just to name a few.

Don't forget finagle and scalding.

At the company I work at, various of the above projects are integral to one system or another (though to be fair, there are quite a number of ex-Twitter folks floating around). I'm not a Scala expert, but as far as I can tell Twitter also has a large footprint in that community.

Don't forget Bootstrap CSS.

Your post implies they made the libraries you listed, but they certainly didn't make Mesos or most of those things.

I think Mesos is the only one from that list they didn't make, they were just an early adopter on that one

You're right, they didn't make it and open source it. But they did work closely on it's creation, as far as I know.

(edited above post to remove Mesos mention).

You're actually right Benjamin Hindman now CEO of Mesosphere Inc. started what became the Apache Mesos project while a PHD student at Berkeley, He later left Berkeley, interned at Google, then took a full time role at Twitter where he built out Mesos at scale.

His linkedIn bio: Member of the Technical Staff Twitter March 2010 – May 2013 (3 years 3 months) Built out Apache Mesos (http://mesos.apache.org).

Storm shouldn't really be on the list fwiw. They bought it and then replaced it with Heron after a couple years but barely worked on Storm in the interim. That was all Nathan.

I think they played a big part in shaping Mesos early on if any of the interviews/talks I've listened to are any indicator. They were one of the early adopters and hired one of the people that created the project at Berkley.

In fact they did purely make most of those things. Also they took Mesos from a barely working prototype to a system that scales to 100K+ nodes running many thousands of different workloads in production. They did that not only by hiring Benjamin and paying him to work on it but also by surrounding him with two of the strongest engineers I've ever worked with (John Sirois and Bill Farner) as well as 3 or 4 more I didn't work with as much so I can't say how much they added but it was clearly a large investment. Effectively they seed funded it for almost 4 years. Other people worked on it too, but they were the lion's share of the commits and did almost all of Aurora too.

Tired of hearing devs reel off these projects like they actually matter. None of that is user facing, none of that is worth the thousands of employees.

Stop looking at everything with dev eyes.

Can't most of those projects be boiled down to dealing with scaling? Twitter has seemed to have been mostly focused inwards, trying to get themselves more efficient at being Twitter. The other companies I mentioned seem more ambitious in expanding their tech outwards.

That's a No True Scotsman perspective. Scaling isn't easy.

I'm not saying it is easy or disparaging the work of the engineers there. I am saying it seems like they have a different approach to how tech fits in with the goals of the company. To oversimplify, Twitter is focused on efficiency while others are focused on growth. Maybe the actual goals are different, but that is the impression I get looking at their output.

Can't reply to the child comment -- but worth pointing out that open sourcing your growth magic would be considered pretty dumb by most/all of SV. In fact open sourcing your scaling magic is considered dumb by most of SV. Finally, Twitter clearly doesn't have growth magic (or they'd be growing faster) -- but is that an engineer's fault? At the end of the day, any user facing engineering is beholden to the product team. Engineers at Twitter can run experiments, but they can't get those experiments shipped unless a PM is behind it. I'm not saying the PM team at Twitter are blocking awesome ideas -- just that there's a real shortage of awesome ideas that have been proven to work in the Twitter context. It's tough.

[full disclosure, I wrote one of the projects mentioned above.]

Reply on news.ycombinator.com