Thought it would be an actual web crawler from the title, got really disappointed.
Good news: it looks like the server-side crawler code that the browser extension sends content to is open source too: https://github.com/mwmbl/crawler-server
Ah ok.
It is an actual web crawler. I should have put a link to the server side code.