Downloading the internet, or how we got our first 1M articles
preona.net
preona.net
You say that you're downloading the entire internet for content, and that you'll be serving it up to your readers directly for the various reasons that you site through your app.
If you're linking to the content, no issues. But this doesn't sound like linking: you're stripping the content, and now you're serving it to your readers yourself, so isn't that taking the content from the provider and using it for your own purposes? Are you only doing this with providers that gave you permission to do so? And if not, is there some sort of fair use clause that you see applies here?
Just very curious as to how you see this model working with copyrighted content (and/or definitely let me know if I'm just missing the model here/misreading what was written in your post...end of a long day! ;-)
The difference between what search engines/the Wayback Machine do and what a service like his does is that the former caches more of less complete copies of the content without the intent to strip anything out (save for the view as text only feature in Google's cache), while his service is stripping out the monetization schemes of the content owners.
Although he links back to them (which he absolutely must do anyway), the content owners might become angry that they aren't getting the traffic and the chance to have their ads clicked. I think that is a reasonable viewpoint for the content owners to have, yet I also want to believe that a service like his should be able to exist. Perhaps some kind of middle ground can be achieved where he does show the ads but perhaps well organized at the bottom or middle of the article or something.
What would the content owners be able to do though? Maybe send DMCA notices. This would fall under copyright infringement laws I think. I'm no lawyer though. I also think there should be some kind of allowances for these kinds of content repurposes.
What about copyright legislation? As a rule of thumb you should assume that you are forbidden to do anything with other peoples content that isn't explicitly allowed.
I know law changes/not everyone sees it the same/different courts rule differently, but if one accepts that taking an article in full and republishing on another site (even if sourced/linked back) without permission is a copyright violation, I'm not sure that you doing the same thing (and on a wider scale) isn't also a copyright violation, even if the final article is presented on a different medium.
I'm so very much not a lawyer (pretty obviously), just someone with a publishing background, but it might be worth you talking with a copyright lawyer just to make sure you're on solid ground before you launch it with the content/way you're currently planning.
Also a million is such a nice milestone, I just had to write a blog about it :)
Since you're on AppEngine does that mean that your code to extract the article content is written in Python? If so, consider using the Python port of Readability.js: https://github.com/srid/readability
I haven't tested this one personally (yet), but before it came out I ported Readability to Python and it was very quick, less than a second to extract the content. I used lxml to parse through the X/HTML. You'll have to do a lot of tweaking to get the heuristics working more or less perfectly but it would be well worth it.
(The following assumes you're downloading and processing the content using AppEngine) As for bandwidth concerns, you will probably want to just get a fat pipe into your place of business (or wherever you're operating from). Where I'm at it's only $100/mo for a 16mbit/down line. That would handle your growth for quite awhile. Once the content is all downloaded and extracted locally you can push it up to your AppEngine server.
As a result, it's better to stick with the javascript version of readability and run it in node.js. It's perhaps a dash slower than a python version would be, but the results are much much better and it's easier to keep up to date.
fiam@raichu:~/printful$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 8
model name : AMD Athlon(tm) XP 2600+
fiam@raichu:~/printful$ python readable.py http://www.nytimes.com/2011/01/25/world/europe/25moscow.html > /dev/null
Took 0.193097171783 seconds for http://www.nytimes.com/2011/01/25/world/europe/25moscow.html
I'd say that hosting something like this on App Engine, which prevents you from using C extensions, is not a very good idea.
Edit: formatting
I think it takes around 0.5 seconds on a PC, a lot of this resulting from a pure javascript implementation of pretty much everything.
Also, the publicly available port of readability to python is ridiculously slow... I was about to make it about 10 times faster by re-implementing it myself.
I can't release the source code, but the idea is to invert the algorithm: do one pass through the DOM collecting stats about children nodes, text length, and link density, instead of repeated queries at each step.
You can get up to 10-20 docs/second on commodity hardware with this approach with an cElementTree DOM and python.
1) you have to pay for it
2) you can be blocked by the content providers
Would it be possible to leverage your end users instead?
While your users read articles in your app, the app probably does little; couldn't it use that (mostly idle) time to scrape other articles, and send back to your central server the scraped/cleaned content? Then all the server does is store and serve.
(Of course you'd have to warn users and check for the type of connection (only do this when the user is on wifi?) but it would be quite legitimate for your app to download content that has actually been requested by the user).
heh, it's always fun launching.