205 karma · joined August 14, 2015
"Correct" my ass
IMO the hardest things in distributed crawling at scale are a good URL frontier, priorities, rate limiting and things like that, which are quite often overlooked.
People are so fragile nowadays on the internets. how come?
About the entertainment aspect. I think the point here is following: modern web media/pics/etc is too attractive and people can't focus on things what really matter. Every website with "funny cats" tries very hard to retain you as user and your attention -> entertain you more. I think many people (myself included) fall in risk group and can't actually measure/control time spent on procrastination while browsing infinite number of subreddits or something for interesting stuff. Even when i really need things to be done - there is something that can immediately capture my attention. And I have to say I am not alone.
The root cause here is inability to focus on important things and finally quit procrastination.