Is there a standard tool that does this? Especially for, shall we say, highly interconnected webpages? I need it for Overcoming Bias.
Write a crawler that examines all local links, and builds a graph (it needn't be anything fancy; a collection of nodes and edges is enough, but you'll have to detect cycles when building the graph), and then print it out in graphviz .dot language.
Then you can feed it into neato [either through a command-line pipe, or construct one using Python's subprocess module], et voilà, instant pagemap! :)
You can also use some of the rendering modes and scrape position data for the rendered nodes, if you want to generate a clickable imagemap.
Output from the above is at: http://jey.kottalam.net/tmp/obgraph.out
I tried rendering this with "dot -Tcmapx -oob.map -Tgif -oob.gif" but dot segfaulted after 70 minutes. The code as posted at gist only outputs nodes for articles written by Eliezer in an attempt to make the dot file a manageable size. Tips/fixes appreciated.
1) Rendering an undirected graph via graphviz's neato tool.
2) Rendering to svg, which might let graphviz not have to keep as much rendering information in memory at any one time.
3) Asking about it on the grapviz-interest mailing list. https://mailman.research.att.com/mailman/listinfo/graphviz-i...
There's got to be standard graphing tools for things that are highly connected...
I guess what he meant was tools which can do this for a particular site.
I can't think of any but as a previous poster said - a crawl of desired depth for all local pages, then feed the access log to "statviz" to generate a neat dot file along with the links.