Ask PG: Karma Data Dump | Hacker News Reader

Ask PG: Karma Data Dump

17 pointschasingsparks28 comments

This might be outside the scope of the Ask HN tag (although it is not really a feature suggestion either), but can you provide an SQL dump of all users Karma (i.e. url, vote sums, and dates). I think there are many people who might be curious about the statistical properties.

View on HN

For the life of me I can't find the sql connection in arc, would you help me find it ;) ?

http://ycombinator.com/arc/arc3.tar

Personally I think releasing that data is a bad idea. It might help people to game the system and it might inadvertently release more than intended, see:

http://arstechnica.com/tech-policy/news/2009/09/your-secrets...

I am not sure the game-the-system argument is strong. I could write a scraper to get the same data I am requesting in about 20 minutes. I cannot speak for other members, but I personally would want a karma dump to see if I can improve the algorithm. It is already pretty robust against abuses, but I think it could be improved as far as ranking goes.

HN has pretty good bot blocking measures, a little while ago someone posted a less-then-optimal script to retrieve the users karma on the fly and almost all of the users of that script got blocked.

Datamining a body of data like that is likely to turn up all kinds of interesting tidbits, amongst others you could use it to identify the 30 or so 'secret' power users on HN, figure out when they're sleeping and so maximize your take on spammed links.

Even now it can take more than 2 hours before a spam link gets deleted, no need to make it worse.

Not everybody uses HN with good intentions, there are plenty of people that see this as just another place to dump their links.

If you're looking for power users, it's public who the 100 users with the most karma are: http://news.ycombinator.com/leaders

Indeed, but using that metric might lead you to conclude that pg and other luminaries are just bit players :)

PG would be number 1 by almost a 2:1 margin if he were on the list (he has 50,000+ karma, vs. 27,000 for nickb). The software explicitly blocks him (and has since HN was released), presumably as a way to make the site feel like a community and not his personal vanity site.

That page doesn't include any admins. The page is defined on line 2258: (newsop leaders () (leaderspage user))

Leaderspage is just below: line 2262. The line we're interested in is line 2266: (each u (firstn nleaders* (leading-users)) This line says "for each of the first nleaders* in the list returned by (leading-users), do the following:".

Leading-users is line 2276. It returns a sorted list of users by karma. When looking at a user and deciding whether to sort em, it asks two questions: "does this user have more karma than the constant leader-threshold*?" and "is this user not an admin?"

So if you ever find a user who has more karma than whoever is the last user on the leaders page, but isn't on it, either you've found an admin, or the code running this site has been changed from the 3.1 release.

There is only one user like that afaik.

Why assume that SQL is involved at all? Hint: it's not.

That was the point.

TSV, JSON, or whatever then -- it's the less relevant detail. I just want a dump and I consider it uncouth to scrape if someone is willing to freely provide a dump -- or at least, I would prefer permission.

It is all flat files. Read the arc code, it is very interesting.

Is it transactional?

I think it's the write-to-a-new-file-every-time kind of transactional. That probably works just fine for this kind of application.

OK, so if your data model depends on multiple files for persistence, there's no transactional behavior?

I am interested in the Parato-ization of up/downvotes. If one recorded all users who visited each news item and whether or not that user voted at least once, what would the distribution of voting percentages look like?

In the US, old people are a powerful voting block because they vote a lot. Do a small fraction of HN readers have a highly disproportionate effect on karma?

There's no database for HN. Everything is saved as text files. The save-votes function (line 133 of news.arc), which writes out the hash table of a user's votes to a file, is as follows:

(def save-votes (u) (save-table (votes* u) (+ votedir* u)))

So the data could be gotten as a zip, but not a SQL dump.

Interesting. If HN ever decides to go the database route, sqlite might be a good fit since it's relatively simple and everything is stored as a single file.

PostgreSQL stores everything in flat files too (on Windows, in a directory like "C:\Program Files\PostgreSQL\8.1\data\base\100929"), and I'm guessing MySQL would too.

Are you nutts!? sqlite LOCKS the entire database for every update!

I think this information would be really cool, maybe you could cluster people together depending on the content of their submissions, to find people whose submissions you could monitor.

That was part of my reasoning. There are many karma algorithms that could be deployed in different contexts. If the HN readers found some good alternatives, maybe PG -- or whoever maintains the sites code -- could integrate it as a user preference. For example, I would prefer to read the comments of people who have a few very strong comments rather than people who have many average comments.

(Note: I am completely ignorant about HN's code -- I don't know how they cache or calculate anything).

By the way, what would happen if all karma was reset?

The front page would be empty until people started upvoting from the new page.

Comments would be ranked purely reverse-chronological until people started voting.

It'd be less disruptive to the front of the site than another revisiting of 'Erlang sure is interesting'. Past threads would slowly be mined for re-karma as they were re-submitted and found in google searches.

    I think there are many people who might be curious about the statistical properties

I am also :)

this would probably be better titled "ask pg". he's probably the only one who can [reasonably] help you.

edited.

I would be one of those people.

Reply on news.ycombinator.com