3,623 karma · joined September 6, 2010
- How many servers (if distributed)?
- How many cores/server?
- What kind of processing takes place for each page?
Does it just download and save the pages somewhere (local filesystem, cloud storage, database) or it extracts (semi) structured data? And so on.
Specifics aside, these days it's not hard to crawl millions of pages/day on commodity servers. Some related posts:http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-bil...
http://blog.semantics3.com/how-we-built-our-almost-distribut...
http://engineering.bloomreach.com/crawling-billions-of-pages...
http://engineering.bloomreach.com/crawling-billions-of-pages...
"""
web chat is often (ab)used in a way that makes it the equivalent of global variables for communication: if offers the path of least resistance for sharing information (state) at the cost of making this information harder to retrieve, search, structure, associate, leverage in the future. Or to try another metaphor, it's the equivalent of dumping data on flat free-text files instead of a relational database.
Web chat does away with two (at least) properties I have come to appreciate in written communication: being asynchronous and contextual. Although modern web chat systems typically work even when you're offline by sending notifications, they're primarily a realtime system. There is the implicit or explicit expectation that a chat message (especially a direct one) should be answered asap, unlike an email or a tracker comment. As for context, there is little to none. In more than one company I've seen chat effectively replace the dedicated bug/ticket tracker system, with people polluting one or more channels with intermixed discussions that would be better off as comments on the ticket at hand. It's a mess for anything other than transient, throwaway stuff you wouldn't mind purging after a few hours or days.
Looking forward for a "web chat considered harmful" blog post if there hasn't been one already.
"""
Web chat does away with two (at least) properties I have come to appreciate in written communication: being asynchronous and contextual. Although modern web chat systems typically work even when you're offline by sending notifications, they're primarily a realtime system. There is the implicit or explicit expectation that a chat message (especially a direct one) should be answered asap, unlike an email or a tracker comment. As for context, there is little to none. In more than one company I've seen chat effectively replace the dedicated bug/ticket tracker system, with people polluting one or more channels with intermixed discussions that would be better off as comments on the ticket at hand. It's a mess for anything other than transient, throwaway stuff you wouldn't mind purging after a few hours or days.
Looking forward for a "web chat considered harmful" blog post if there hasn't been one already. </rant>
I guess promising to (not) do `X` and not keeping your word makes you untrustworthy but what is the literal harm on someone other than the self-referential belief that breaking promises is bad? To illustrate, let's assume that `X` is something like "I promise to not eat zucchini without you ever again". How is "cheating" by secretly gorging once on zucchini with some friends harmful to the person you give it to, other than their self-induced idea that it is bad?
alias rstrip='sed -i "s/[ \t]*$//"'
# delete .pyc files alias delpyc='find . -type f -name "*.pyc" -delete'Can you list a few examples of such flaws?
That's a bold assertion, bolder than "most people who work at startups usually don't get to have competing offers for $250k to pass up"