A new algorithm predicts which Twitter topics will trend hours in advance | Hacker News Reader

A new algorithm predicts which Twitter topics will trend hours in advance

web.mit.edu

110 pointsdenzil_correa25 comments

This seems like a great trending topic detection algorithm, but nothing revolutionary. The "hours in advance" part is probably more to do with the scale of twitter's data and their decisions about when to publish topics on their website than it does the novelty of the approach.

It seems like a great and tested method for developing your own trending topics algorithm, which could be useful in other contexts. I wonder if this could be applied to corporate email servers to let management spot issues before they're brought up.

This is something I would spend money on. Does anyone know if such an Exchange/Gmail plugin exists?

Does the algorithm still work when it includes it's own effects?

If an algorithm like this was widely used, the predicted topics will trend just because they have been picked -- the algorithm will fulfill the prophecy itself.

Well, you just answered your question, didn’t you?

Exactly. I was also thinking that Finacial Engineering also tries to garner the advantage of statistical analysis of stock market based on previous trends. But I am sure there is no 100% accurate model. Moreover statistical model need to be changed as behavior start changing.

I wonder if that will reflect here as well.

In any case, It has some interesting applications as they mentioned in ticketing.

The only concern I have is the way the experiment is conducted. The correct way to collect data would be to consume the Streaming API to collect the random sample of tweets provided by Twitter [0]. They should then run their algorithm to compare how they are doing against Twitter's algorithm. Currently, they are filtering out certain topics of interest and just comparing these topics. Therefore, they are filtering a great degree of noise which is actually observed by Twitter.

[0] https://dev.twitter.com/docs/api/1.1/get/statuses/sample

Thanks for the interest! We collected tweets by sampling a small percentage of all tweets in a time window, to emulate what one might get from the streaming API (I did this as part of the VI-A http://vi-a.mit.edu/ masters program at MIT, and was an employee of Twitter). We did pick a fixed set of topics to track, but those were randomly sampled (though we did get rid of topics that trended multiple times in a large time window, like the name of a football player who scored in multiple matches, in which case we don't know which event we are trying to detect). One thing the algorithm doesn't do at the moment is come up with its own trending topics. It just tests prediction of trending/non-trending on a hold-out set taken from the original set of topics.

So this is an algorithm to determine what another algorithm will determine will be popular? Meta.

Do I predict this sparking a restriction in data availability in the API?

That was my initial thoughts. Might this be a potential way to monetize twitter finally?

Could you elaborate please? Are you refering to the "early access" to trending topics, or restricting limits on API?

Funnily enough, Twitter could already sell 'early access' without any new algorithms. Just time shift the availability of results based on payment status. So for £100, see things that are guaranteed to be trending next hour, because they're technically trending now, but we're withholding it.

They are already monetizing their data. DataSift is one of their key official partners [0].

[0] http://datasift.com/

I'm bookmarking this to read later, but how much more sophisticated does the algorithm have to be than:

1) Create a list of 500 - 1000 active, relatively popular Twitter users: this would eliminate most celebrities who only tweet casually or delegate it to their PR people...presumably, by the time they tweet something, it's already huge.

2) Segregate the sample group of Twitter users into cliques

3) When any topic spreads between multiple cliques at an accelerated rate, that topic will likely trend

In addition, have a list of mega-popular celebrities and assume that most of what they tweet has a high probability of being a trending topic.

Twitter has some kind of formula for removing constantly-popular topics (or else Justin Bieber would forever be on the list)...if there's an easy way to include that, then it seems like predicting trending would be straightforward?

I agree with you --- this seems like a perfectly good way to tackle the problem of trend prediction directly. What we had in mind was something that would be more generally applicable to any kind of time series data, and we figured it would be interesting to test it on Twitter trends.

Do you think a Twitter trend which is a meme like #ReplaceAMovieTitleWithBed will be detected using this scheme?

The algorithm itself seems very simple. The paper quotes "The Unreasonable Effectiveness of Data". The surprising thing is the simple algorithm works great given large data.

Is it easy for academic researchers to get firehose access?

Not in general. I was part of the VI-A program (http://vi-a.mit.edu/) at MIT, which allows you to do your thesis at a company.

I think I could use this to gain tremendous karma on reddit...

Just watch out for the upcoming trend #Earthquake

That's what the Italian scientists did.

can it be used to predict market trends? instead of topics use stock tickers?

There is already some work done on this front [0]. Ruiz et al basically concluded the stock price of a company is directly proportional to the number of discussion topics of that company viz. if there for a company X there are fewer topics under discussion the stock prices would be poor.

[0] http://www.cs.ucr.edu/~vagelis/publications/wsdm2012-microbl...

Thank you for sharing this.

Reply on news.ycombinator.com