Hacker News Reader

Comment by kharms | Hacker News Reader

This would be fantastic. Twitter's utility is in its open (inherently less profitable) nature. As a data source for academic study it's unparalleled.

View on HN

It's unparalleled in that there is very little that's even similar out there.

I'm less convinced of its actual value. The Twitter-based papers I've read seem to all have two things in common: First, they all use Twitter data. Second, they all seem to desperately want to present observations about how people interact on Twitter as being universal, even though nobody seems to be able to come up with a theoretical or empirical reason why that should be the case that's more compelling than, "We we have to pretend it is in order to get published."

Similar like in Sublevel?

There's thousands of sites serving the same or similar purpose. Twitter is only unique because it has a lot of users.

It's not really about whether or not the site works similarly to Twitter. It's about the neat stuff people can build with something like Twitter data. I don't think you could do something like http://blog.idl.ssrc.msstate.edu/?p=42 with Plurk data and get anywhere near the same kind of attention for it.

For that kind of thing, having lots of users is the key feature of interest.

Is attention the goal? I've been struck by the number of articles and research projects that are some version of "we got lots of tweets and graphed them" and have totally failed to go the "so what" step.

Why should we care about this project? If you want to use this to find power outages using Twitter, why not just do that directly with e.g. grep (also, more importantly, are these outages about which the utility company doesn't already know?). Having big data (related: 12 Million lines of text isn't big) isn't cool in and of itself. Doing things which help with that data would be cool.

Yes. It also bothers me that quality news sources, like NPR, have fallen into the trap of presenting Twitter as representative of the general population.

They often report about how some group of people on Twitter are angry about something -- as if that matters. Reddit has almost as many monthly active users lately, yet no journalist makes the mistake of reporting on Reddit comments as if it's generalizable to the larger population.

Most of the news reporting I've seen cast on individual Reddit communities. I don't think such a thing can be done as easily as twitter except through hashtags, thus the coverage on twitter being more twitter-wide hashtagging rather than sub-tweet communities.

What's interesting is that, even though they're not well-defined, there certainly are specific communities on Twitter. There's tech, and sports, and science, psychology, politics, music, etc, and each of those can be further divided. I'm sure that if Twitter put in the work, they could suss out all these communities based on who is following who and what they tweet about. (In fact, I'd be surprised if they aren't already doing that.) They could then make these communities accessible to the public.

Twitter currently has lists, but the problem is that these lists aren't crowdsourced (to my knowledge). They're maintained by individuals, which is unsustainable, because one person can only do so much work keeping a list accurate and up-to-date.

I'd be pretty sure that "Who to Follow" is based on some sort of social graph though probably in a simplistic way, e.g. lots of people you follow follow this person and you don't.

Yeah, that's the only thing I've seen from them in this domain. It's just woefully lacking imo. I don't want to pick people to follow individually... I want to pick entire communities of people to follow.

So essentially lists created using social graph algorithms. I probably wouldn't use it personally but I can see how it could be useful.

> I probably wouldn't use it personally but I can see how it could be useful

This describes virtually every announcement I've heard from twitter.

Twitter is a cheap way to get response quotes for an article without asking anyone any questions. It's gotten so ridiculous that some news "content" is just a series of embedded Tweets. Serious news organizations should band together and restrain themselves from this silly practice. It violates one of the most important rules of interviewing, which is to know who the source is and what her biases might be so that the quote can be presented appropriately.

There's a lot of lazy journalism looking for a quote, any quote, because they're required to have one in a story they're writing.

This isn't new. When I was an analyst, some reporters would call me for a quote about something. There were times when I didn't have anything to say because I didn't know anything about said thing. But some would pester me for a quote anyway so they didn't have to give someone else a call.

Twitter does seem to have become a particular crutch though.

The BBC was one of the most egregious pioneers in soliciting meaningless viewer responses in the pre-social media era. They would go so far as to kill 30+ seconds of airtime to soliciting/demanding "participatory" viewer responses that were invariably inane. On the plus side, at least it led to some good satire [1].

[1] https://www.youtube.com/watch?v=OQnd5ilKx2Y

Auntie Beeb's insistence on such complete drivel has killed all the joy and interest. I find it utterly unlistenable much the time.

yet no journalist makes the mistake of reporting on Reddit comments as if it's generalizable to the larger population.

You should stop by /r/austin sometime, and just lurk. The amount of local reporters looking for some kind of pulse to put their finger on and even those nu-'journalist' sites (Vice,Vox,insert here) obviously mining for quotebait is absurd, and the community has been calling it out.

Of course, /r/austin isn't a remarkably interesting place to be anyway for a lot of reasons...it's just interesting to watch the "Austin" phenomenon manifested there. Different story though.

  quality news sources, like NPR

NPR is CNN marketed towards psuedo-intellectuals by replacing common terms with sophisticated sounding vocabulary. Most of the pieces that discuss topics I am heavily familiar with are shallow and often misrepresented.

I say this as someone who really wants to like NPR and struggles to find any unbiased news source.

I'm sympathetic to claims of NPR bias (still a big fan), but rather than seeking an "unbiased" source, I want a thoughtful, reasonable, open-minded and well-mannered conservative news source. It does not exist.

You might look into the magazine The American Conservative. It's far from perfect (Pat Buchanan uses it as his soapbox from time to time, for starters), but on the whole it's pretty readable.

I heard one of the "big data social scientists" to claim that they can now replace the population with the sample (as this is what the Twitter really is) because of, you know, big data.

If you have a demographic model of twitter users and appropriately re-weight, why is this a bad approach to take?

If I understood correctly then this was considered as a not necessary step.

Now given the Twitters nature then is there such a model that is even approximately accurate?

Or in other words, what relevant data Twitters collects and how accurate it is?

Because you don't (and probably can't) know that this model is accurate? Also, there are a lot of populations which are not present on Twitter at all, and more which are not present in meaningful numbers.

Such models presumably can be tested in some sense. E.g. build some sort of predictive model of elections in each state using Twitter users as a surrogate, see how well you do.

I'm not claiming that such a model is guaranteed to be perfect or even as good as a pollster, but it sounds in principle like a reasonable thing to do, if you take appropriate precautions to reduce bias.

At least one can try definitely.

>As a data source for academic study it's unparalleled.

Does that actually mean it's valuable? Just because something's one of a kind doesn't mean it's worth that much.

The phrase "x is unparalleled" implies uniqueness by virtue of the value it provides.

GP is basically saying twitter has accumulated a vast wealth of social data that is immensely valuable to the research community.

Sounds like the academic community should be charged for access to the data, if there is so much value in there.

For full access to the twitter archive, they are, though a lot of data can be gleaned from what's publicly posted.

In retrospect I should have assumed that to be the case

The media is clearly benefiting, with many stories backed by a Donald Trump tweet. But if the tweets weren't free to begin with, they might not be as newsworthy.

> the media is clearly benefiting

Journalists certainly are. Getting quotes was hard work. Now it consists of trawling Twitter. Not sure if this has been a boon or bane for journalism, however.

> Journalists certainly are

In the short term, anyway. The substitution of Twitter-trawling for actual journalism and research is doing immense damage in the long run.

I think you (and many others) have this relationship by the wrong end. Things which are valuable are frequently rare/unique, whereas there are many unique/rare things with no value whatsoever. As others in this thread have implied, the lack of ability to monetize is strongly correlated with a lack of real value.

that is my concern as well, if it truly was so widespread and useful I doubt they could turn a profit, however its real market is apparently much smaller than people suspect.

twitter simply cannot reach people who don't go looking for it, it would need to basically just randomly show up on your instant messenger unbidden, it has no universal connector like radio and tv which are everywhere.

then the last issue is there are too many voices on twitter so only those connected get out and assume importance. importance to only users of the medium

> twitter simply cannot reach people who don't go looking for it

I'm not so sure. Hashtags are everwhere both physical and digital, companies want to to start tweeting about their event/product/etc. As for a direct connection, how often have you see a tweet embedded in a news article, or displayed on live TV?

Hashtags are everywhere, but it doesn't follow that Twitter is - see FB, Insta, even LinkedIn.

It means it is easy to access. That is all.

Maybe they can pair up with Elsevier to charge outrageous subscription fees for access to the data to help it become profitable.

Depending on the kind of non-profit Twitter tries to become, they may need to somehow disentangle Twitter from all of the political favoritism and censorship controls currently built into the system.

Seems like a pretty huge change and it's definitely tough to envision!

Bonus: non-profits can still make tons of money for their founders and operators. People think it means 'no profit' but that is totally off base.

Operators, yes – they can get typical compensation. But a non-profit founder doesn't get anything.

As a non-profit it could also become a decentralized platform, and it could more easily fight-off censorship requests.

Reply on news.ycombinator.com