Crawl Order and Disorder
marginalia.nu
marginalia.nu
Separate the problematic domains and run them in a lower priority background crawl; then do the opposite as well, have a higher priority list for crawling sites that have regular updates more often, based on their last update gap.
Basically like a generational garbage collector but in this case it's generational content collector, although arguably, still mostly garbage.
Am I correct in assuming you are recrawling your entire corpus to get fresh results? Would there be a downside to replacing crawl epochs with a continuous crawl that is random but with age priority?
Seems the bigger problem is very large domains with slow response times and long crawl delays.
> Am I correct in assuming you are recrawling your entire corpus to get fresh results?
For known links, I'm sampling them and based on whether I find changes (first via if-none-match and if-modified-since or alternatively via locality sensitive hashing), I'm recrawling only a part or all of the links.
> Would there be a downside to replacing crawl epochs with a continuous crawl that is random but with age priority?
The drawbacks to this is a much more mutable crawl data (being able to read or write crawl data top down in an append-only format is a huge performance improvement), as well as problems with the indexing software which takes ~1 day to complete, and can't currently be partially rebuilt but is rebuilt from scratch every time at significant computational expense.
https://www.youtube.com/watch?v=EDfm2fVS4Bo
A noob question, is Elixir good at crawling tasks in the process simplying the system architecture.
A crawler, and especially a very large one like Marginalia, is going to implement its own queuing logic, its own retry logic, its own management of same, and so on and so forth. As a result, things like "BEAM has supervisor trees" are actually not that useful; as they are nowhere near enough for this use case and need to be augmented anyhow, they don't really help a lot, and can even get in the way. At most the BEAM OTP infrastructure might help you bootstrap something up somewhat more quickly but I'd expect single-digit weeks into the dev process before it isn't really that helpful anymore. None of the BEAM-specific strengths strike me as hugely helpful here, mostly for similar reasons; they don't quite match what a crawler wants per se and the crawler is going to reimplement them anyhow.
For a project of this scale, you also don't want to be doing the raw indexing in Elixir, as it is not a very fast language, and at this scale that adds up quickly, so you're going to pull in another language anyhow.
All in all, while I might call it "acceptable", there's a solid half-dozen languages (plus runtimes, as appropriate) that I'd put solidly in front of Elixir for this use case and another several I'd rate as roughly ties. It certainly is not the case that it offers some sort of amazing, blow-me-away advantage that makes it the only sensible answer or anything.
The hard problems in crawling is mostly dealing with a very large and potentially highly mutable state. It's a concurrent problem, but a fairly easy one and a good use case for a traditional thread pool since there is latency is mostly a non-issue.