Comment by giberson | Hacker News Reader

Comment by giberson | Hacker News Reader

If I weren't too timid to risk doing so, I would do the following (read I hope someone else does this).

Process the pdf's with an OCR program to extract as much text from each document as possible. The extraction should be done page by page, so the extracted text can be referenced to a PDF page#.

Then, provide a searchable/browse-able directory of the extracted content. Each page of text has link to the original PDF page so you can easily open up the PDF to the page the text was extracted from.

I'd also make all text user editable wiki style. Combined with the inline PDF page references it would be super easy for any user to fix up translation errors from the OCR process. Tie in a karma system to the users profile so that edits can be thanked/kudos on a job well done to help with automating moderation of user edits by rating the user's current karma to decide if the edit should be accepted automatically or provided as an alternate version other users can check and rate up if they think it should replace current version.

Maybe mash in an image cropping service so diagrams can be cropped from the PDF and inserted inline with the translated text. Provide simple wiki formatting markup to allow users to format the articles.

Use ad revenue/donations to alleviate/cover hosting costs.

1, 2, 3, go.

Well, most of this already exists in Google books, where I'm reading some of these particular scientific journals right now, can switch to OCRed text at any time (albeit with ftrange contemporarie fpelling), or download a facsimile of any public domain work in PDF format. The only thing they don't have in place (and should add) is the wiki-style crowd editing.

So the guy has basically built a 33gb torrent of stuff that was already freely available to the public, just from a different source.

DocumentCloud [1] provides all the pieces necessary. It's all very nicely documented, too.

[1] http://www.documentcloud.org

That is what jstor is about actually: full text search of a number of articles.

I haven't used it seriously in quite a while, but I remember the search being fairly primitive, though faceted.

They don't do full text search on these articles.

There's already a system somewhat like that in place and it was actually referenced in the article. Wikisource (http://wikisource.org) is run by the same nonprofit as Wikipedia, and has a pretty robust system for pairing up side by side transcribed texts and scans. Someone's already started the germ of a Phil. Trans. library (http://en.wikisource.org/wiki/Philosophical_Transactions). At present the first volume is done and about forty volumes (http://en.wikisource.org/wiki/Index:Philosophical_Transactio...) are uploaded and ready for users to transcribe.

If you're curious about what these papers are, here is a hack at exposing the metadata in an easily searchable format. It would not be hard to scrape the PDFs to searchable text using unix tools (doesn't come out very readable) and render them as SVG files with inkscape for easy browser viewing:

http://robocracy.org/royal-search/

https://github.com/bnewbold/royal-search

Lots of good old stuff, such as "On the Double Organs of Generation of the Lamprey, the Conger Eel, the Common Eel, the Barnacle, and Earth Worm, Which Impregnate Themselves; Though the Last from Copulating, Appear Mutually to Impregnate One Another"

This is all available elsewhere, i'm just trying to demonstrate how low the barrier to search is with OSS tools; I hope all these papers end up on archive.org with full text search!

I used OCR to do some interesting things with a similar dataset:

http://news.ycombinator.com/item?id=2791611

Distributed Proofreaders already has some infrastructure set up for proofreading and editing OCR'd documents. It's not quite the same as your proposal, but they do track stats and have levels of participants based on skill and experience. Might be worth checking out.

   http://www.pgdp.net/c/

Regarding this, since the archives are in the public domain, wouldn't it be possible for any third party (e.g. google books) to ask and obtain permission to photograph the originals ?

Anyway the ethical thing to do here is to lobby jstor (a non-profit organization) to make reprints free.

Reply on news.ycombinator.com