Hacker News Reader

Comment by amix | Hacker News Reader

While Twitter a is cool service I can't imagine what 1000 engineers can do all day long? Based on this maybe they are building their organization wrongly and following their footsteps isn't the correct way to follow.

View on HN

Come on. We're supposed to doubt the merits of this entire article about Twitter's philosophy and general large-organization engineering practices because someone outside can't imagine how so many people fill their day? Because maybe a thing which we can't hope to gain an accurate perspective of indicates they're doing everything wrong?

This reminds me of the story of a walk from SF to LA illustrating why software estimates are always so wrong:

http://www.quora.com/Engineering-Management/Why-are-software...

Instead shouldn't we just evaluate the content on its own merits?

Look at e.g. Instagram or imgur or reddit or HN for a lean team. It's hard to compare without intimate knowledge of what either team faces, but without considering alternatives, many things that shouldn't look plausible actually do, e.g. The $5,000 hammers the army supposedly bought, the (incorrect, but oft repeated) space pen story, or the clusterfuck that is US healthcare.

If you've worked on online software you might have some appreciation of the masses of unseen services that are essential for running the system. All web apps, except in the early days, are icebergs. Payments, analytics, control panels, monitoring, deployment etc. -- there is just a huge amount of stuff that goes on that the user typically doesn't see.

Their monitoring dev team alone is 22 [1], and they specifically mention decomposition of services as you say.

[1] https://blog.twitter.com/2013/observability-at-twitter

I have actually built and scaled a system similar to Twitter called Plurk (which has millions of users, billions of data items and similar use cases as Twitter). We built this with a team of a few developers and one sys-admin. While Twitter is at a much larger scale, I really doubt you need 1000 engineers to solve that problem, because even at Plurk's scale we built automated systems for scaling, monitoring, backups etc. (e.g. sharded the databases, used master-master MySQL setups that supported auto-failovers, nagios for alerts etc.)

You may say: but they have mobile apps? But so do a lot of other companies that don't have 1000 engineers. You need a few really great mobile devs to build great mobile apps, again based on my experience.

I think what has happened for them is that their structure isn't that good and they have scaled too fast (and in this process they have hired a lot of B and C people that aren't that good). It's a least my conclusion based on the limited information I know about them.

I for one would be (honestly) interested in reading your article in the same vein as the original - what would you say is and is not necessary in his take?

Take a look at https://github.com/twitter this should give some insight into where they may be spending their time.

There are a few projects that would be well known (Storm, Kestrel, Bootstrap, FlockDB). Although I'm 100% if they were created at Twitter or just hired in.

If Plurk has 5 developers serving 1 million daily users, and Twitter has 1000 developers serving 100 million daily active users, you only have to believe that scaling by 2 orders of magnitude is twice as hard for it to be a tie in terms of developer:user ratio.

(I suspect the write load is rather different - the figures I can find for Plurk suggest 8 billion messages in total - Twitter writes half a billion a day.)

The big advantage of software though is that you don't have to maintain a fixed ratio of developer:user like that, though. Sure, your employee count should increase as user count does, but far less than linearly.

He suggests that it might scale logarithmically, not linearly.

I'm not sure this is in practice true. Twitter have to start writing or tuning their own scale-specific software, need more analytics, application security, developer productivity, partner and sales engineering etc. proportionally than they would do at 1% of their size. Certainly they could run 100 Plurks with many fewer than 1000 engineers but I'm not sure that's the best way to characterize Twitter.

I don't think there's a huge differences serving 1 million daily users or serving 100 million. In both cases you need highly scaleable systems. In 100 million case you need a lot more computers, but I think the overall structure of the systems could be very similar (e.g. sharded databases, auto fail-overs, extensive monitoring etc.) Of course you need a larger team serving 100million users, but I don't think you need to scale the team as much as Twitter has done.

one of the problems there is that everything is extremely convoluted.... every single thing is a huge rat's nest of cross team dependencies. I found it an extremely frustrating environment to try and work in.

This is pretty common in the growth phase of a company, especially one with easily scalable demand such as a software company.

As others have pointed out, a lot of projects have been built from 'scratch' due to Twitter's needs and/or culture. This alone has a heavy carrying cost, as they now have the burden of having at least two maintainers for mission critical applications and services. It doesn't mean the engineer-tuples are bijective onto the various projects, but there are only so many tickets a person can resolve.

At Amazon, we grew to what we then thought was an insane size in IT (and across the company – we had horrid little offices all over the place in Seattle). This was boosted by a number of acquisitions, such as drugstore.com and homegrocer.com. In 2000, we had a small round of layoffs, then they had another round in 2001.

I would expect something similar at Twitter if they don't move horizontally. I would say that, given the competitive HR landscape, it's quite possible they intentionally overhired to secure strong engineers and will rank and yank those who have not turned out to be performers.

They're still in an annealing phase, but, honestly, I'm unqualified to speculate on their internal alignment, growth plans, or any RIFs they have planned in the near future.

This is a bit of an unfair request, but it would be interesting to have you break down the engineering groups you think exist at Twitter and estimate their size.

But think what they have the power to do.

They probably make awesome unit test scripts.

Reply on news.ycombinator.com