Hacker News Reader

Comment by slg | Hacker News Reader

Was Twitter ever really about tech excellence? My perspective as an outsider who doesn't live or work in Silicon Valley, it always seemed more of a communications and media company founded on a tech backbone. Beyond their early scaling problems, which they spent a long, long time fighting, what really exciting tech has emerged from Twitter? Google, Apple, Amazon, and Facebook all have much more impressive technology to show for themselves over the last several years.

View on HN

Twitter's Github: https://github.com/twitter

Apache Aurora, Bootstrap, Storm, Heron, Pants, ThriftMux, DistributedLog, Apache Cotton, Iago, just to name a few.

Can't most of those projects be boiled down to dealing with scaling? Twitter has seemed to have been mostly focused inwards, trying to get themselves more efficient at being Twitter. The other companies I mentioned seem more ambitious in expanding their tech outwards.

That's a No True Scotsman perspective. Scaling isn't easy.

I'm not saying it is easy or disparaging the work of the engineers there. I am saying it seems like they have a different approach to how tech fits in with the goals of the company. To oversimplify, Twitter is focused on efficiency while others are focused on growth. Maybe the actual goals are different, but that is the impression I get looking at their output.

Can't reply to the child comment -- but worth pointing out that open sourcing your growth magic would be considered pretty dumb by most/all of SV. In fact open sourcing your scaling magic is considered dumb by most of SV. Finally, Twitter clearly doesn't have growth magic (or they'd be growing faster) -- but is that an engineer's fault? At the end of the day, any user facing engineering is beholden to the product team. Engineers at Twitter can run experiments, but they can't get those experiments shipped unless a PM is behind it. I'm not saying the PM team at Twitter are blocking awesome ideas -- just that there's a real shortage of awesome ideas that have been proven to work in the Twitter context. It's tough.

[full disclosure, I wrote one of the projects mentioned above.]

Don't forget finagle and scalding.

At the company I work at, various of the above projects are integral to one system or another (though to be fair, there are quite a number of ex-Twitter folks floating around). I'm not a Scala expert, but as far as I can tell Twitter also has a large footprint in that community.

Don't forget Bootstrap CSS.

Your post implies they made the libraries you listed, but they certainly didn't make Mesos or most of those things.

I think Mesos is the only one from that list they didn't make, they were just an early adopter on that one

You're right, they didn't make it and open source it. But they did work closely on it's creation, as far as I know.

(edited above post to remove Mesos mention).

You're actually right Benjamin Hindman now CEO of Mesosphere Inc. started what became the Apache Mesos project while a PHD student at Berkeley, He later left Berkeley, interned at Google, then took a full time role at Twitter where he built out Mesos at scale.

His linkedIn bio: Member of the Technical Staff Twitter March 2010 – May 2013 (3 years 3 months) Built out Apache Mesos (http://mesos.apache.org).

Storm shouldn't really be on the list fwiw. They bought it and then replaced it with Heron after a couple years but barely worked on Storm in the interim. That was all Nathan.

I think they played a big part in shaping Mesos early on if any of the interviews/talks I've listened to are any indicator. They were one of the early adopters and hired one of the people that created the project at Berkley.

In fact they did purely make most of those things. Also they took Mesos from a barely working prototype to a system that scales to 100K+ nodes running many thousands of different workloads in production. They did that not only by hiring Benjamin and paying him to work on it but also by surrounding him with two of the strongest engineers I've ever worked with (John Sirois and Bill Farner) as well as 3 or 4 more I didn't work with as much so I can't say how much they added but it was clearly a large investment. Effectively they seed funded it for almost 4 years. Other people worked on it too, but they were the lion's share of the commits and did almost all of Aurora too.

Tired of hearing devs reel off these projects like they actually matter. None of that is user facing, none of that is worth the thousands of employees.

Stop looking at everything with dev eyes.

My impression is they have an excess of technical talent. They have a huge engineering staff, and from n outside perspective, I'm not convinced their product demands it.

Google, Apple, Amazon, etc., all have much more interesting products, which in turn gives their technical staff many more interesting problems to solve.

The common advice here is "let Twitter be Twitter," i.e., they don't need to expand into new products, let alone crack VR, or self-driving cars, or drone deliveries. That path will need to accompanied by major restructuring.

"Tech excellence" sounds like such a useless buzzword. There is no such thing, just incrementally better.

Everyone I've met who was a techie there has been one of the best I've encountered. I know everyone here thinks Twitter is a fancy rails site with some Kafka in the back, or whatever, but they actually do have some tough problems to solve. Or did, once upon a time.

Reply on news.ycombinator.com