Hacker News Reader

Comment by noelwelsh | Hacker News Reader

If you've worked on online software you might have some appreciation of the masses of unseen services that are essential for running the system. All web apps, except in the early days, are icebergs. Payments, analytics, control panels, monitoring, deployment etc. -- there is just a huge amount of stuff that goes on that the user typically doesn't see.

View on HN

I have actually built and scaled a system similar to Twitter called Plurk (which has millions of users, billions of data items and similar use cases as Twitter). We built this with a team of a few developers and one sys-admin. While Twitter is at a much larger scale, I really doubt you need 1000 engineers to solve that problem, because even at Plurk's scale we built automated systems for scaling, monitoring, backups etc. (e.g. sharded the databases, used master-master MySQL setups that supported auto-failovers, nagios for alerts etc.)

You may say: but they have mobile apps? But so do a lot of other companies that don't have 1000 engineers. You need a few really great mobile devs to build great mobile apps, again based on my experience.

I think what has happened for them is that their structure isn't that good and they have scaled too fast (and in this process they have hired a lot of B and C people that aren't that good). It's a least my conclusion based on the limited information I know about them.

I for one would be (honestly) interested in reading your article in the same vein as the original - what would you say is and is not necessary in his take?

Take a look at https://github.com/twitter this should give some insight into where they may be spending their time.

There are a few projects that would be well known (Storm, Kestrel, Bootstrap, FlockDB). Although I'm 100% if they were created at Twitter or just hired in.

If Plurk has 5 developers serving 1 million daily users, and Twitter has 1000 developers serving 100 million daily active users, you only have to believe that scaling by 2 orders of magnitude is twice as hard for it to be a tie in terms of developer:user ratio.

(I suspect the write load is rather different - the figures I can find for Plurk suggest 8 billion messages in total - Twitter writes half a billion a day.)

The big advantage of software though is that you don't have to maintain a fixed ratio of developer:user like that, though. Sure, your employee count should increase as user count does, but far less than linearly.

He suggests that it might scale logarithmically, not linearly.

I'm not sure this is in practice true. Twitter have to start writing or tuning their own scale-specific software, need more analytics, application security, developer productivity, partner and sales engineering etc. proportionally than they would do at 1% of their size. Certainly they could run 100 Plurks with many fewer than 1000 engineers but I'm not sure that's the best way to characterize Twitter.

I don't think there's a huge differences serving 1 million daily users or serving 100 million. In both cases you need highly scaleable systems. In 100 million case you need a lot more computers, but I think the overall structure of the systems could be very similar (e.g. sharded databases, auto fail-overs, extensive monitoring etc.) Of course you need a larger team serving 100million users, but I don't think you need to scale the team as much as Twitter has done.

one of the problems there is that everything is extremely convoluted.... every single thing is a huge rat's nest of cross team dependencies. I found it an extremely frustrating environment to try and work in.

Their monitoring dev team alone is 22 [1], and they specifically mention decomposition of services as you say.

[1] https://blog.twitter.com/2013/observability-at-twitter

Reply on news.ycombinator.com