Simplifying Systems with Elixir • Sasa Juric • YOW! 2020
https://www.youtube.com/watch?v=EDfm2fVS4Bo
A noob question, is Elixir good at crawling tasks in the process simplying the system architecture.
https://www.youtube.com/watch?v=EDfm2fVS4Bo
A noob question, is Elixir good at crawling tasks in the process simplying the system architecture.
A crawler, and especially a very large one like Marginalia, is going to implement its own queuing logic, its own retry logic, its own management of same, and so on and so forth. As a result, things like "BEAM has supervisor trees" are actually not that useful; as they are nowhere near enough for this use case and need to be augmented anyhow, they don't really help a lot, and can even get in the way. At most the BEAM OTP infrastructure might help you bootstrap something up somewhat more quickly but I'd expect single-digit weeks into the dev process before it isn't really that helpful anymore. None of the BEAM-specific strengths strike me as hugely helpful here, mostly for similar reasons; they don't quite match what a crawler wants per se and the crawler is going to reimplement them anyhow.
For a project of this scale, you also don't want to be doing the raw indexing in Elixir, as it is not a very fast language, and at this scale that adds up quickly, so you're going to pull in another language anyhow.
All in all, while I might call it "acceptable", there's a solid half-dozen languages (plus runtimes, as appropriate) that I'd put solidly in front of Elixir for this use case and another several I'd rate as roughly ties. It certainly is not the case that it offers some sort of amazing, blow-me-away advantage that makes it the only sensible answer or anything.
The hard problems in crawling is mostly dealing with a very large and potentially highly mutable state. It's a concurrent problem, but a fairly easy one and a good use case for a traditional thread pool since there is latency is mostly a non-issue.