The text density-based Go crawler
Introducing the text density-based Go crawler.
Using these metrics for a web crawler are an interesting take.
You essentially just count the number of words per "area" of text (plus the ratio of linked words to all words), and you can pretty much tell the "main content" apart from navigation elements, without actually having to render the page. It's not perfect but it works at web scale; see my WSDM 2010 paper, "Boilerplate Detection using Shallow Text Features", for example.
There are different ways to define "area" for density, and it looks like this implementation computes the density for an entire subtree of DOM nodes. It's interesting that no matter how one defines "area", it will work somewhat well either way.
Fun fact, you could say that I got my PhD thanks to my bad eyesight. Some time around 2008, I browsed the Web without my glasses and I noticed that some gray areas are denser than others...
My interpretation of the situation is that there is a fundamental text law at play here, with two types of text, "long text" (anything as long as a sentence and longer) and "short text" (things like navigational menus, buttons, anything that needs to be quickly understood, like "STOP", etc.), and that also applies to non-Web printed material (travel catalogs, for example).
If you are curious about this statistical text law, you'll find more details in my PhD thesis, https://www.repo.uni-hannover.de/bitstream/handle/123456789/...
I can't quite grok what is going on, but referencing the paper gives a great starting point. Thank you.
In particular the notion of a DensitySum seems novel and worth exploring.
(for others, http://www.ofey.me/papers/cetd-sigir11.pdf)