Hacker News Reader

Comment by greghinch | Hacker News Reader

> By the end of the year I’ve already promised this, so the engineers – when I promised it publicly they’re already mad at me so they can keep being mad at me.

Wow I would hate to work at Twitter.

View on HN

Without knowing what's actually going on internally that's just a snap judgement. If the engineers are actually being pressured to achieve it in too short a time then sure, however it's quite likely that "mad at [him]" means in a friendly way, and he's going "would love it if you can prove me right, otherwise don't worry, I'm already telling the press that I was an idiot who spoke without checking with you, so if we're late then it's not a problem".

I've never worked for a company where executives and/or salespeople didn't promise features that don't exist without first consulting engineering.

Really? I know this is a subtly different thing but is it possible to name a company that has always delivered what it said it would when it said it would? Apple would be my nearest candidate, but they have messed up here.

I'd be really curious to know more about Twitter's database setup. There's clearly something that they've done for scale that makes this a difficult thing. Otherwise, implementing a button that SELECT * FROM tweets WHERE user_id = 1; wouldn't take long to implement.

Twitter moved on from having a single table of tweets a long long time ago. I imagine archived tweets (which aren't necessarily that old) are spread out in multiple archive databases and perhaps not even in traditional db formats. And I imagine that, as a rule, no user facing code is allowed to touch any of the databases, but rather goes through services with heavy caching. So that a seemingly simple feature is an engineering challenge really isn't that surprising. Twitter is optimized for very recent content.

I can only guess the format for archived tweets is very different from newer one, maybe just a compressed text file, which the engineers have to automate extracting and parsing at scale. Doesn't seem that hard, but the format was probably a hack job back in the day and there are probably a lot of considerations, e.g. metadata like RTs and favorites.

Plus, CEO's comments sound like it's been set up as a side project. The kind of management style books like Mythical Man Month and Peopleware warned us about.

Indeed, this would also explain why the search function only allows one to view tweets going back seven days.

I interviewed with a Twitter search engineer and he told me that that's because their in-memory search index is not big enough hold all tweets ever posted. On top of that, their efforts to scale up the index are counteracted by the ever-increasing volume of tweets posted per day.

And the reason it's not big enough is because they don't want to invest in it. They found out search is not as monetizable as they thought it would be, and at the same time they cannot take away the functionality. The compromise is to keep what they have.

Lots of guessing, hypothesizing, and imagineering for information that is readily available. Here's a start:

http://engineering.twitter.com/2012/04/mysql-at-twitter.html

http://highscalability.com/blog/2011/12/19/how-twitter-store...

http://www.percona.com/live/mysql-conference-2012/sessions/g...

http://www.slideshare.net/yousukehara/introduction-of-twitte...

http://engineering.twitter.com/2010/05/introducing-flockdb.h...

http://www.infoq.com/news/2009/06/Twitter-Architecture

http://engineering.twitter.com/2010/05/introducing-flockdb.h...

https://github.com/twitter

The disdain for developers is probably not restricted to third parties.

Reply on news.ycombinator.com