What is "next gen" in this implementation? Chrome support?
IMO the hardest things in distributed crawling at scale are a good URL frontier, priorities, rate limiting and things like that, which are quite often overlooked.
IMO the hardest things in distributed crawling at scale are a good URL frontier, priorities, rate limiting and things like that, which are quite often overlooked.
Fast Ethernet is my favorite example.
[0] (translated title)
(Planck length is 10e-35. Even the strong nuclear force operates on a scale that's like 20 orders or magnitude larger (10e-15). And a hugantuan electron? Forget about it.)
Could contact me? We may have some interests in common. Check my profile.