YCRFS 3: Things Built on Twitter | Hacker News Reader

YCRFS 3: Things Built on Twitter

ycombinator.com

81 pointspg68 comments

https://web.archive.org/web/20091101135023/http://ycombinato...

The beauty of twitter isn't that you send messages without specifying recipients. Any billboard does this. HTML webpages, blogs with RSS feeds, etc do this.

Twitter allows you to:

1. Communicate on equal footing. You're living on someone else's blog, or their social news site, or their subreddit, or their messageboard, or in their chat room. There's no need to shout anyone down with negative karma, because everyone controls their own feed.

2. Control your world. If you stop following someone, they basically cease to exist. On the other hand, they can follow you if they want, with no harm to you. This allows celebrities to share directly in a way they never have before.

3. Stay engaged with minimal time. Tweets are limited to 140 characters for everyone. Writing a blog post requires a commitment and willpower. A tweet is extremely easy.

Despite all of this, Twitter itself is very awkward to use, and making sense of the un-threaded mass is very difficult. It's reaching mass adoption because it can be all things to all people (microblog, chat, rss feed, friend updates, broadcast announcements, etc).

There are better ways out there to facilitate good discussions. Google is trying to tackle the office collaboration problem with Wave, and I think there's huge progress to be made in the space of online discussions, separate from the current plethora of news reaction sites (coughhackernewscough) that we have now. The inevitable rise of self-posts on news sites (it's usually resisted or frowned on at first) shows the existence of this need.

I agree that the great advantage of twitter compared to other social tools is that you don't need both parties consent for it to work. Anyone if free to listen to you without requiring your consent.

It is more like in real life, where I don't need to ask your permission to listen to what you say.

This isn't really related to the RFS, but the idea of Twitter as a new protocol sort of scares me. Most of the successful internet/web protocols so far have not been controlled by a single company, but if Twitter can sustain their current growth there's no reason for them to open up the "protocol".

Spot on! My exact thoughts. It would be hard to think of Twitter as a protocol since nothing about Twitter is open except for their API. Google wave would be a more appropriate example for a web service/protocol.

Directly related to this is the BRDFDR/FETHR distributed microblogging protocol.

For background and rationale, see the Birds of a FETHR talk slides & audio:

http://brdfdr.com/pres/

twitter is one of the new programming ecosystems that devs will chose to play in. like fb or aws or gae or iphone, etc. note nasdaq recently released an iphone app that utilizes stocktwits which in turn is built on twitter.

Well, SSH was controlled by a single company at first...

But maybe calling Twitter a "internet protocol" is a bit of a stretch, even in purely technological terms? Human communication protocol, maybe...

just to paraphrase Stephen Colbert: "The protocol we didn't know we needed until we had it"?

A lot of Twitter traffic is machines now. Actually, this is making it a little hard for humans, since there are so many spammers/pornbots.

And there're are a bunch of examples of protocols that were initially controlled by single companies. AIM gave us Oscar and TOC. Microsoft gave us SMB. Kazaa gave us FastTrack.

The difference however is that SMB, for example, was implemented by Microsoft but can run on any machine. Twitter is tied to twitter machines.

I think "messages without a recipient" have been around for quite a while, and we call them broadcast messages. DHCP does this. Radio does this. Blogs do this. I'm not saying that Twitter doesn't have an interesting innovation here (I personally think they do), but I don't think they've "discovered a new protocol". They've taken existing concepts and applied them to people.

> I think "messages without a recipient" have been around for quite a while, and we call them broadcast messages.

Yeah or any number of pubsub, queue based messaging systems. What twitter has is millions of users all in one place with low barrier to get started and easy access via mobile, web, and API. It has its warts, but so do many popular things (TCP/IP for example), the thing that matters about it is that it is deployed and used heavily and in people's consciousness.

I'll take this opportunity to plug my site that is built using the Twitter API - http://www.playtwabble.com . Twabble is a vocabulary game based on the words you use in your Twitter status updates. It uses a point system based on the game Scrabble. I built it to learn Django and Python. I didn't get much feedback when I submitted it to HN, so here goes again? Hope this isn't spammy, I think my game illustrates a basic example of what is possible with Twitter.

Let me also plug my site :) http://www.celebsutra.com It aggregates tweets and pictures from celebrities on twitter. Had submitted it to twitter and have implemented most of the changes people had suggested.

Let me do the same, http://twitthegame.com/ It's a place to talk about sports on twitter (like stocktwits).

Should we be doing this startup the Twitter-way, or do we need a business plan too?

No business plan needed. All you need to know is roughly what you want to do.

pg: Is it okay to contact you with questions? If so, where? Thanks.

Sure: pg@ycombinator.com

Pretty sure that having a fully formulated business plan isn't what YC is generally pushing for anyway based on history.

Does anyone know what the current rate of tweets per second is? This would influence what it's possible to do with the streams.

Right now about 400 per second.

http://www.twitpocalypse.com/

You can read every tweet from a single machine. It's just a matter of getting acces to the steaming API.

You can read every tweet from a single machine.

Sure, but the rate of tweets will determine how much processing you can do per tweet given a certain amount of computing power.

The data input bottleneck is small. If you're going to do something in n*log(n), you should be fine. If you're going something in n^2, you're fucked anyway. Map Reduce is your friend. On that note, I only learned about Hive since coming to facebook. They made and open sourced it. Check it out: http://wiki.apache.org/hadoop/Hive

If you manage to save them somehow (or distribute them to different machines), you can distribute the processing, I suppose.

Exactly -- thus the question of how many machines would be needed to perform a certain amount of processing per tweet, which ultimately comes down to my original question of how many tweets there are.

I have decent access to the track/follow apis. I just queue the tweet right away and have a bunch of workers take care of it. Something like rabbitmq works wonderfully here.

This is cool.

I've been trying to figure out Twitter for the last couple of years. Left to the defaults, using twitter.com as the "interface" it seems to be a giant attention-whoring spew of marketeers and SEO "experts".

To me (and I think I said this a long time ago in a post here) the potential as a global distributed message bus always seemed so much more powerful. It's kind of like IRC to the next level.

I'm going to have to think on this. I've been toying around with some Twitter-based thoughts relating to my home automation and video analytics hobby-projects.

The key to Twitter is to think of it as a big old stream of data.

Of course you don't want to read it all. That's why you follow people. It pulls their information out of the stream, and makes a substream for you. The same goes with turning on people to mobile: it pulls them out of that stream, and into one even closer to you.

I hear you, but I'm not so sure that it's just about picking people to follow. That feels too yesterday... I'd like to pull trends out of the conversations at large. Can Twitter be a trending engine? Can you leverage a topic for financial gain? Is there actionable data in the random blatherings of a million people?

If I am JetBlue (I'm not) and people are ranting or raving about some experience on Delta, can I use that data? Run a promo? Give them a safe haven?

In one respect, Twitter seems like it can be the exposing of the raw conscious of "the public", that group that has been so elusive to marketeers. The trick is to extract the data, to measure the thing, without affecting that which you are measuring. One of the longest standing challenges in science.

I'm rambling, but thinking on iterations of this in my head.

If you have any suggestions to improve our tool, we'd be happy to hear them.

http://trendistic.com/

Twitter most certainly can be a trending engine. Twitter.com actually has trends on the right hand side, but there are tons of sites that do other things with trends, too.

As for JetBlue/Delta, I've bitched about Verizon on Twitter, and within about an hour, had a follow and an @reply from my local division's Twitter account. They then followed up with phone calls, and took care of my problem.

And now I tell you that story, which also furthers their image...

With proper filtering, a news alert system could be pretty cool coming from the firehose. I'm thinking identifying key words like the Moldavian #pman tag with regard to the student shutdown of the gov't following alleged voter fraud. You'd need to establish a solid baseline for a large number of words to discover variations from the baseline. carpdiem and I have discussed this one for a bit, but we couldn't figure out how to finagle access to the data stream to start developing. Could be interesting...

Maybe going beyond simple wordcount to determining statistical probabilities of certain words / phrases being mentioned together & at "high" frequencies would taper down the number of words you need a baseline for (because I also don't assume the baseline will be static or even near stable - changing trends of discussion topics)

This idea also ties into the "new news" idea a bit. What we've seen from events like #iranelection is that twitter is a good (but could be great) tool for detecting news at the onset of events (before big stations pick up on it), but really starts to get bogged down with "spam" or RT's once a topic gets picked up. I'd like to see news that integrates these abnormal spikes in tweets that hint at a bigger underlying story w/some sort of follow-up with more context about the event.

Since we're talking about Twitter being a "protocol"...Is it likely they will be the only 140 character microblogging platform? What opinions do you have on Status.net and what they are doing with www.identi.ca? I think twitter has created a new protocol, but not a monopoly on that protocol. In 5 years people will be microblogging everywhere. That's what worries me about building something off twitter.

@PG If only our 'other ideas' is a response to a RFS, would you like us to check the 'this application is a response to a RFS'?

No, though if it's an alternative idea you're really serious about you could file a second application for it under one of your cofounders' accounts.

Twitter is the “Atari Pong” of real-time messaging services?

I remember how exciting it was back in February when I started realizing what it meant to have a real-time messaging protocol. http://www.thinksketchdesign.com/2009/02/25/design/algorithm...

How magical! Imagine all the strange and wonderful things we could do with it - like water our plants automatically! But I'm going to have to guess that twitter has seen its heyday. Twitter has been fueled by the exciting idea of a real-time protocol - and that idea will truly shape the next generation of the internet. But that idea is here now and it's going to outgrow twitter in a hurry. The idea of a real-time protocol is going to be the backbone of the next internet-merged-with-mobile-cellular-satellite-and-wireless-mesh-network revolution. Every device will talk directly to any other device. The ISPs will be the next media giants of this year - scrambling to hold onto their market models as the internet as we know it dissolves into the fabric of the technological landscape powered by open source network communication software and ad-hoc device-to-device mesh networks. Twitter has been a real inspiration, but I just don't see them growing and adapting to be the driving force to carry this revolution. Thanks, twitter, but I'm looking to google wave now as a protocol that has real potential to be the backbone structure for the next-gen internet. Or hey - prove me wrong. It definitely seems like a good move to reach out for fresh ideas at ycombinator.

More on google wave - and annotated video clips that summarize the google wave demo video: http://www.thinksketchdesign.com/2009/06/09/web/media/google...

[offtopic-ish] This is the first decent short summary of twitter that I fully get. I haven't "bought in to it" yet, due to my free time being fairly limited, and to me it doesn't seem to add much value compared to the time I think i need to spend to get the hang of it... maybe I'm just getting old...

ok so here's my idea:

you take the data out of the hose and run a bunch of "machine learning" over it to figure out what twitterverse is saying about apple. or rather, apple stock.

if you can guess the opening direction the next day better than 50% of the time i'd say you would have some money on your hands.

i kid, i kid...

we're actually doing something similar to this, but have been tracking a number of stocks as well as other things.

twitter + sentiment analysis = prediction market; but in most observable cases, we're seeing twitter lag behind the market, not actually predict it. oh well!

Hey, any chance you have raw data to process? I'd like to give it a shot (doing sentiment analysis on your data)

Our internal data is kinda one of the things we curate and I can't really give that away, but I do have access to an academic corpus that might be redistributable. I'll have a look. :)

You don't have to be right in the market 50% of the time. In fact you aren't guaranteed to be profitable if you are right 99% of the time.

Your winners just need to be bigger than your losers.

Average winner profit * winning percent > average losing loss * loss percent.

in any case, check out http://stocktwits.com I think they even have access to the real time feed.

What's more, you can even go broke with overall positive expectancy, if your position sizing strategy can't handle losing streaks.

actually, dont kid.

Lets assume Information propagates according to some diffusion model, forward only in time, and after some delay affects the price of a given stock. Twitter is the perfect feed to match against high frequency stock data to measure each stocks sensitivity to particular kinds of news (location, keyword)

So you build a weighted model of what kind of news affects which stocks, and how strongly.. and do algorithmic trading on that statistical arbitrage, and exploit the market not being perfect [thus improving market efficiency].

The engine you get is quite similar to one which acts as a real-time agent looking for twitter activity on topics of interest to you...

Why kid? It seems like a plausible/testable thing.

this is a sure loser based on efficient markets as this information is already priced in.

The markets aren't perfect.

But, if an algorithm can find this mispriced asset, and trade to exploit it before a human can, then that should improve the markets efficiency and reduce the arbitrage.. or so it goes.

What is the lag of twitter compared to paid news services? Probably enough to make investing in building such system totally unattractive compared to aforementioned paid feeds.

So you'll have to make your trading decisions based on sentiment or even insider info... Very risky and not very profitable proposition...

Not that twitter is totally useless for trading, but not very useful for HFT uses you describe.

I wonder how the rest of us mortals get firehose access. We could do some pretty cool stuff with it, and it'd negate the need to absolutely hammer the Search API (er.. not that we found workarounds that involve hammering the Search API.. oh no).

Use what they give you for free. Impress them. Ask for better access. Profit!

What twitter NEEDS is a web service that allows you separate the real humans from the bots and not-real accounts. Various applications can then be built on top of that information.

From personal experience using facebook and twitter, facebook is a much richer source of information to be mined, to bad though that facebook is for the most part a closed platform, at least on the level that you get with twitter being able to analyse the stream of tweets or searching all tweets made.

Wonder if people have created facebook apps for the sole purpose of getting access to a lot of users data then on sold that access to others? If that's even possible.

I'm going to be applying under this. The only problem is that I just left my former startup about a week ago. Looking for someone who would be interested in applying along with myself. Someone who is an awesome developer with a strong machine/statisical learning background prefered. Please e-mail me at kyleATcodeincarnateDOTcom.

Dear [deity],

Please have someone make a noise-filtering system!

What would be left?

While some of the aggregate information that can be gleaned form twitter is interesting, I have yet to see anything approaching a single meaningful message.

I know that many people seem to like twitter, but every time I go to the twitter website I leave without experiencing anything interesting. Maybe I'm doing it wrong.

Maybe I'm doing it wrong

I use Twitter extensively, but I can't say I've gone to the actual Twitter website more than a dozen times. I think the most interesting uses of Twitter involve using Twitter data in other places, like putting a stream on your blog. I think that's where the future of Twitter is. I saw a cool Twitter/Flickr mashup a while back that took a Twitter stream and translated it into pictures, which was cool.

We are on it; http://thecadmus.com.

Hey jayair,

Cool service - I just tried it out on my account. I'm not really sure if I see a huge increase in relevance, although that might be because I follow a lot of random accounts vs. people I actually know. Feedback: I don't like that it's a blackbox and I can't scale/finetune things to adjust the filter. What gets filtered out should be transparent if you want people to rely on that service.

Me is the best noise-filtering system for me that I have seen.

Also you have to define noise, and why are you getting it anyway?

Is there a public dataset of tweets?

There was one available for download, but it seems Twitter pulled it: http://blog.infochimps.org/2008/12/29/massive-scrape-of-twit...

I think the future of twitter is in effectively tracking what genuine people need, want, and feel and displaying that information in an accessible manner. I have no idea how to execute that thought but it seems like the next logical step.

All the "genuine people" I know in my life don't use Twitter and have no intention of doing so. Not one of them needs to tell people they don't know how much they love apples.

http://twitter.com/#search?q=i+love+apples

All people will simply STOP using twitter.

I used to be on Twitter often, and so were many people in my small city. Now, if I go back, about 2 or three people I followed post to twitter every couple of weeks (if at all) where there used to be about 30. They all just stopped.

It's like ANY new toy. You play with it for a few weeks right after Christmas, but then, it just ends up in the toy box and is never played with again.

People though the same about the web in general, but things change when you build around this infrastructure. The first time I saw the web I though nobody would stay much time around a computer to look at crappy web sites. But the really good web sites came in, and we are here 15 years later.

Twitter is not a protocol. This Pubsubhubbubtub-thingy is.

So, who's going to make the next spymaster?

Reply on news.ycombinator.com