Comment by nostrademons | Hacker News Reader

Comment by nostrademons | Hacker News Reader

Parent Full thread

I had a similar interview question for Google. I was given a graph of both average user latency and server load, where average user latency was inversely correlated with server load, i.e. latency went up when the load on the server was at its lowest and vice versa, and was then asked to brainstorm reasons why this could be the case.

The actual answer (I could tell this was drawn from a real experience) was "China". The valleys in server load corresponded to EST nights, which happens to correspond to the workday in China. During these time periods, there are fewer users online, but the vast majority of them are located in Asia, where a response to them needs to get through the Great Firewall of China and cross a trans-Pacific cable. Meanwhile, the U.S. west coast is just going to bed and the U.S. east coast & Europe are sleeping, so all of the low-latency users drop out of the population sample. It was a nifty application of both Simpson's Paradox, correlation-is-not-causation, and speed-of-light limits.

Extra points if you can think of how to avoid strange debugging situations like this in the future.

I had an actual high severity issue on-call page over this kind of thing once.

When I first started with Amazon, I was a dev on the team owning the UI component that all warehouse Picking associates used. Had to run on mobile handscanners with IE5.5. The UI was dead simple, pretty darn fast and that speed matters for worker efficiency. Taking 3 seconds between an item scan and the next pick being displayed was unacceptably slow.

One night a few weeks after Christmas, I got an autocut alarm at around 10pm. It said "PickUI page loads 90th percentile > 5 seconds" or something to that effect. I logged in and sure enough, the latency was spiking up and down like crazy for the entire European stack. All our dependencies were rock solid, metrics showed stable performance. Network was fine, no issues. Rendering was normal. But the round trip latency from the users' perspective (sent via Ajax post after a page load) was showing some round trips were taking 15 seconds!

After 30 minutes of scratching my head, I happened to accidentally switch my graph from "P90 latency" to "number of data points". There were very, very few. It turned out that since the Christmas season was just over, all the EU wearhouses were only running day shifts that week. (That doesn't happen very often anymore, I hear). In the entire European region, there was one picking associate working. And she was picking in some area with poor WiFi signal. Around 20% of the time, her page loads were slow because her WiFi sucked. I drew out a rough map of the locations she picked from and the latency- they was some kind of dead zone. But she was the entirety of my metrics for Europe, so that meant my 90th percentile was terrible.

I modified the alarm to have a minimum number of data points threshold and went to bed.

Took me a couple times to read through this to realize that this was asking about the intermediate trip time between the server and the client rather than the total round-trip time that it took to process requests, including processing time.

Very creative question, requires thinking outside the box. I don't think I would be able to answer it correctly.

As measured it was actually talking about total round-trip time including server processing, but it was measured from within the JS on the client, from start-of-AJAX-request to finish, and so it included both server/client hops in addition to processing time. That's part of the red herring: since you expect the bottleneck to be server processing time, you look there first.

A decent partial answer to "How to prevent this?" is to look at additional metrics - graphing server-only latency would show the expected correlation with server load and an inverse correlation with total RTT, and would narrow the search space significantly.

> The actual answer (I could tell this was drawn from a real experience) was "China"

“China” would have been my first answer to that question.

After spending so much time in a project that deals with massive amounts of HTTP requests from China, I realized that the majority of web developers are making too many assumptions. Several times I have found myself patching pieces of code because the connection was suddenly dropping in the middle of the operation, so I had to add retries and read incomplete streams of bytes to recover at least pieces of the actual data from our Chinese netizens.

In retrospective, I have learned a lot about TCP in the last year with this project.

It's extra fun when you're debugging strange connection resets that occur on a seemingly-random subset of A/B tests, and then find out that it's because a substring of the digits in the experiment ID happens to match the numeric date of the Tiananmen Square massacre, which is censored by the Great Firewall.

https://en.wikipedia.org/wiki/Illegal_number

I just tried https://www.microsoft.com/19890604 from inside China (Wuhan, to be specific) on a naked (non-back-doored) connection. I got Microsoft's 404, not any of the GFW behaviour I've come to know and loathe.

I think I'll be calling bullshit on this.

And another data point.

https://www.baidu.com/s?wd=19890604&rsv_spt=1&rsv_iqid=0xfd2...

No blocking there and it's a Chinese search engine!

Read vermilingua's Wikipedia link on Illegal Numbers. This was in 2012, which lines up with the reference cited on Wikipedia for the GFW blocking the numbers 89, 6, and 4. They reversed course after getting called out for this being brain dead.

To clarify, I actually queried http://www.microsoft.com/19890604 and had it redirected to https://... by Microsoft's server. So the URL query was in plaintext.

Just, you know, so the undies come unbunched before people comment.

Several months ago, I was doing ping geolocation on a bunch of VPN servers. To see if they're actually located where claimed. I looked at several AirVPN servers, claimed to be in Hong Kong. Based on data from hundreds of probes, they did in fact seem to be in Hong Kong.

However, minimum(rtt) was ~300 msec for probes in several mainland locations, some <100 km from Hong Kong. Why might that have been?

What kind of position were you applying for that they asked you this very interesting question?

Frontend software engineer. Latency management is included in front-end engineer's job (because most of the time, it's Javascript's fault), so debugging something like this would not fall outside of the job description. My interviewer was also a frontend software engineer. Both of us ended up doing more general-SWE/TL jobs later on, though (by coincidence, I ended up on the same team as her a year and a half later).

I doubt that they would ask the same question now - for one, most Chinese queries are served out of datacenters in Southeast Asia now (at least when I left; if I believe the Project Dragonfly news I hear about on HN they may be served from inside China), so there's no trans-Pacific cable hop. The Chinese population has also gotten a lot more access to broadband & high-bandwidth mobile connections in the decade since this interview.

Reply on news.ycombinator.com