Hacker News Reader

Comment by thevivekpandey | Hacker News Reader

"It is OK if a query is slow as long as it is always slow" I find this enlightening. I think it is just one instance of difference in emphasis between Google and Facebook. Google, a technology oriented company, would minimize the average response time, while Facebook, a people oriented company would minimize the unpredictability (by minimizing the variance)

View on HN

For what it's worth, Jeff Dean of Google has famously emphasized 95th and 99th percentile performance in preference to average or median performance over the years. The realization that edge cases more powerfully determine user perception of performance than average cases is a deep one, and is not original to Facebook or Google; Dean and colleagues were making some of the same points when they were at DEC's CRC pre-Google.

Do you know of any research supporting that? It sounds plausible to me but a quick search turned up nothing.

There is some social psychology and operations research around this -- not necessarily as it relates to human/machine interactions, but certainly as it relates to such quasi-analogous situations as call centers, lines, speed vs. variability of service at McDonald's, and so forth. The premise being practiced by Facebook takes a lot of cues from behavioral science data that is generalizable to many different domains.

McDonald's, to give a famous example, proved analytically a few years back that -- beyond a certain threshold of commonly expected service times -- their customers would rather get semi-slow service on a consistent basis than highly variable service. In a perfect world, of course, average wait times are as short as possible in addition to a minimization of variance. But when you're at the level of acceptability, there are diminishing returns on speed increases and increasing returns on reduced variance.

Do you have a link to this research or study from McDonalds?

Alas, no, it was from an analytics consulting engagement a buddy of mine did a few years ago. But I am reasonably sure the outcome of the data is public knowledge and available somewhere. I wish I knew where.

Alas, my understanding of this point is entirely folkloric. Self-experimentation, and working on improving real interactive systems, agrees with it. Please update if you find anything more comprehensive.

I think the more compelling reason for "queries must be fast or slow, never 'both, depending'" is that it prevents engineers from accidentally using code originally built for reporting purposes (which might well have no latency requirements, or very relaxed ones) and putting it in a widget which gets slapped on people's home pages. You'd notice on your local, test, or staging instance "Hmm, my home page is taking 10 seconds to load -- that's bad" prior to pushing it live and, whoopsie, there's now 50 million people hitting a very scarily expensive query all at once.

I'm sorry, but your interpretation here is completely wrong.

Facebook (and Google!) cares about performance variance because it has more of an impact on overall site performance than average performance does. Variable performance has huge impact on downstream systems, and you can quickly end up with cascading performance problems.

I think that quote is slightly misleading without more context. They prioritize optimizing variable-performing queries higher than others. They aren't going to be using slow queries on the Facebook home page.

I beg to differ with "Google doesn't care about variances, because it's a tech company and not a people company."

From what I hear, Google/Bing do track response latencies at 99+ percentile.

>> "It is OK if a query is slow as long as it is always slow"

I'm having trouble understanding the motivation. If a slow query is always slow, then I'm always going to be kept waiting for that page/data. It seems logical to worry about the queries that 100% of the time keeps users waiting rather than the queries that keep users waiting <100% of the time.

Does anyone care to explain why this is a good idea (for Facebook at least)?

I'm sure the rationale here is that if a given query takes 100ms, they don't focus on getting it down to 50ms even if that's several times what the average is because they know they can. Certainly someone should focus on making that query faster, but it's more straightforward.

The harder problem is figuring out why that 20ms query suddenly balloons to 200ms. You can say, "no big deal, it only happens 1% of the time," but if you don't know why, you could make changes to the system that cause it to happen much more frequently and eventually bring the whole system down.

Also, there's a bit of UX here. People are much more frustrated by things they don't understand and/or aren't used. There are parts of GMail that are always slow (archiving a lot of messages). I know this so I know I have to wait 5-10 seconds. What if sometimes it took 1 second and sometimes it took 20 seconds? What if it took 20 seconds 5% of the time. I'd probably always click again and think something was broken. If it's always slow, I want it to be faster, but at least I know what to expect.

It's all about user experience. If a feature loads for you in 300ms, then it will feel unbearably slow if it later take 700ms. If the feature always loaded at 700ms, your expectation of 700ms would never be disappointed. The theory is that a consistent 700ms is more appealing than an inconsistent 300-700ms, even though the median is better in terms of raw performance.

A query that is always slow will not end up on home.php. A query that is slow 1% of the time might.

The parent hints at this by talking about minimizing unpredictability and describing Facebook as "people oriented" - it ultimately boils down to user experience.

Think of it this way: if you know a certain function will reliably take a little while to complete, you can justify the effort of adding progress indicators and other feedback to let the user know.

But if query performance is unpredictable, even planning the UI design becomes difficult - not to mention the end user's experience.

Reply on news.ycombinator.com