Hacker News Reader

Comment by djoldman | Hacker News Reader

> With Prequal we saw dramatic improvements across all metrics [for youtube], including reductions of 2x in tail latency, 5-10x in tail RIF, 10-20% in tail memory usage, 2x in tail CPU utilization, and a near-elimination of errors due to load imbalance. In addition to meeting SLOs, this also allowed us to significantly raise the utilization targets that govern how much traffic we are willing to send to each datacenter, thereby saving significant resources.

This feels like one of those "my company realized savings greater than my entire career's expected compensation."

View on HN

The question you should ask is for how much less others would accept to do it. How much it saves isn't how to price some work, you price work based on the people available that can do it.

If the fire department puts out a fire in your house you don't pay them the cost of the building. You don't give your life to a doctor, etc. That way of thinking is weird.

It's not weird, but incomplete. It is broadly captured in the Economics concept of "willingness to pay", loosely the maximum price someone or some firm would be willing to pay for something of benefit to them.

This contrasts with "willingness to accept", loosely the minimum compensation someone or some firm would accept to produce a good or service (or accept some negative thing).

Neither of these is sufficient to determine the price of something precisely, but, in aggregate, these concepts bound the market price for some good or service.

I agree I wasn't super precise, I thought it was enough to show that pricing work equal or close to equal to value produced is unlikely as long as there's many people available to do it.

In my example of the fire department if nobody is really coming and you have no insurance or other way to save stuff you would indeed pay a lot. From what I read these were the dynamics in Roman times.

If you're thinking of Marcus Crassus, the dynamic there was that he'd offer to buy your burning property from you at a steep discount, and would only put the fire out if you agreed.

> "willingness to pay" [and] "willingness to accept"

Often abbreviated somewhat sloppily into demand and supply.

which should make the employee proud, and I'm sure Google is compensating them very, very well.

let's also not forget that the people involved didn't create this in a vacuum. it cost Google a LOT more than their compensation to make it possible for them to even start working on this project, let alone carrying it forward to completion.

people underestimate how hard and expensive it is to manage a company in a way that allows its employees to do a good job.

> it cost Google a LOT more than their compensation to make it possible for them to even start working on this project

The actual "not a vacuum" context here is an environment that has been basically printing money for google for the last twenty years. It did not "cost" them anything. It's fine to acknowledge that the people who built google, however well paid they are, are creating vastly more value than they are personally receiving.

If it costs nothing, why don't we both get together and build a money-printing machine as well?

You say like it's something easy anyone can get done. Have you ever tried? If you try to build a sustainable and successful business, you'll see how hard it is.

You probably know, if you think about it, that I wasn’t saying there were no expenditures involved. This is just specious.

You said “it cost Google a LOT more than their compensation to make it possible for them to even start working on this project, let alone carrying it forward to completion.” As if Google is groaning under the weight of sacrifices made specifically so these SREs can play in their engineering playground every day. I am saying this is backwards — that there has been no such sacrifice on Google’s part.

> Google is groaning under the weight of sacrifices made specifically so these SREs can play in their engineering playground every day

they're literally doing that. google has +$300B net positive assets. [1] they could distribute that to stockholders, go to the bahamas and call it a day. but they keep this wealth there so that SWEs can "play" and create stuff for others to use.

yes, they're not a humanitatian org. they do it so that they earn more, then invest more. but they are sacrificing short term gains and risking actually losing money.

[1] https://www.google.com/finance/quote/GOOG:NASDAQ?hl=en

> but they keep this wealth there so that SWEs can "play" and create stuff for others to use.

That is not their motive or even a secondary consideration for shareholders.

To be fair, Google didn’t found YouTube.

And Youtube wouldn't last long on its own.

Regarding infra, they were not capable of reaching the scale they needed and were already struggling by the time they got acquired.

On the money side, they couldn't attract advertisers as well as Google.

My point was that for whatever reason(s), Google chose to buy YouTube, while continuing to work on Google Video (which was started the month before YouTube), instead of buying a different competitor.

Given that YouTube was founded by 3 PayPal alums and was angel investor funded while being one of the top most visited sites at the time of its acquisition by Google, I am inclined to believe that their issues you mentioned above and others not mentioned would have been resolved in-house, via their partnership with Google Video or another company, or via acquisition by another company.

I don’t think it’s likely that YouTube would have failed as a company with the founders and investors it had, such as Sequoia, as it was simply too popular. YouTube had already started running ads for ~9 months before acquisition, though they were initially against pre-roll and other site-wide ads.

https://en.wikipedia.org/wiki/History_of_YouTube

> Participatory video ads were designed to link specific promotions to specific channels rather than advertising on the entire platform at once. When the ads were introduced, in August 2006, YouTube CEO Chad Hurley rejected the idea of expanding into areas of advertising seen as less user-friendly at the time, saying, "we think there are better ways for people to engage with brands than forcing them to watch a commercial before seeing content. You could ask anyone on the net if they enjoy that experience and they'd probably say no." However, YouTube began running in-video ads in August 2007, with preroll ads introduced in 2008.

Profit margins matter a hell of a lot to publicly traded companies. Share price is a multiple of earnings.

> let's also not forget that the people involved didn't create this in a vacuum. it cost Google a LOT more than their compensation to make it possible for them to even start working on this project

...let's also not forget that Google didn't manufacture this opportunity in a vacuum. It cost the rest of society a lot more than their revenue to make it possible for them to employ people who can spend their entire days working on software.

Didn't it cost society to raise us to adulthood so that we can work for Google and other businesses? How many people have sacrificed their time and attention, directly or indirectly, to our benefit, since we were born? Yet, nobody's saying our salaries aren't deserved or the merit of our own efforts.

I think many people are explicitly saying that.

Pretty common really. Huge web properties like this have just massive room for cost optimizations, and AFAICT, is largely bottlenecked by management capacity. On the other hand, it's also pretty common to emphasize pet metrics like tails, which are pretty much by definition a minority of the cost. We give them the benefit of the doubt because a) google and b) peer review publication, but thats not usually a heurestic available to decision makers.

Optimizing tail latency isn't about saving fleet cost. It's about reducing the chance of one tail latency event ruining a page view, when a page view incurs dozens of backend requests.

The more requests you have, the higher the chance one of them hits a tail. So the overall latency a user sees is largely dependent on a) number of requests b) tail latency of each event.

This method improves the tail latency for ALL supported services, in a generic way. That's multiplicative impact across all services, from a user perspective.

Presumably, the number of requests is harder to reduce if they're all required for the business.

The last sentence of the abstract indicates the goal is to be able to run at higher utilization without sacrificing latency. The outcome of that is you need less hardware for a given load which for Google undoubtedly means massive cost savings and higher profitability.

   > Prequal has dramatically decreased tail latency, error rates, and resource 
   > use, enabling YouTube and other production systems at Google to run at much 
   > higher utilization.

> Optimizing tail latency isn't about saving fleet cost.

Indirectly, it is. As the quote I replied to suggests, in order to combat tail latency services often run with surplus capacity. This is just a fundamental tradeoff between the two variables mentioned.

So, by improving the LB algo, they (and anyone, really) can reduce the surplus needed to meet any specific SLO.

You're correct - that's a fair point.

The person chose to be employed instead of starting their own business. Less risk, less reward.

I disagree with toenail being downvoted. In Europe, employees enjoy a great deal of stability. In Netherlands, it's nigh-impossible to fire someone: you have to fill in 10 pages of justification paperwork and have an independent government agency review and approve it. If someone has a long-term illness then you have to pay 70% of their salary for up to 2 years, even when they do no work at all. Most people don't want to be entrepreneur: they want clear instructions and stability. When you try to give stock to employees, the tax authorities raise an eyebrow: why would you give stock to employees when they enjoy none of your risks? It makes no sense, so we'll treat it as a form of salary, so we'll tax you 52% on the stock's paper value.

At the end of the day, what's left for the entrepreneur? You enjoy all the risk, but you don't get to have a paid 2 year sick leave. Even sympathy for your hard work can be hard to get. The potential of money is all you have.

Things are different in the US of course, where people can be fired the next minute without reason. That looks like just borderline abuse to me. But from a European perspective, the above comment does not deserve downvoting at all.

> In Netherlands, it's nigh-impossible to fire someone: you have to fill in 10 pages of justification paperwork and have an independent government agency review and approve it.

Practically speaking, won't they fire someone by negotiating a "voluntary" severance agreement? It's not like an employee wants to stay on for long once it's been made known that they are unwanted.

Though obviously a severance agreement is better than at-will employment, it's also not a guarantee of long employment.

That is possible, but you'll have to go through a lawyer to draft a contract. It costs time and money (apart from the severance fee). Assuming negotiations are successful. It's still not easy.

What you get as an employee is certainty that your compensation will only increase in the single digits per year and that the pension fund will compound a part of your income in single digits as well (pension funds typically underperform the S&P; even in bad years).

So either 2 years of “stable income” or the chance of much higher compounding rate. As I see it, employment is a nice backup if being self-employed doesn’t work out.

> why would you give stock to employees when they enjoy none of your risks? It makes no sense, so we'll treat it as a form of salary, so we'll tax you 52% on the stock's paper value.

The Netherlands is the only country in the EU that taxes on unrealised gains.

Not sure if you mean just for stock options, but more countries/regions have some form wealth tax on unrealised gains.

Just stock, but I forgot that Spain also have stock included in wealth tax for people with over 700k assets. I think it’s just NL and ES though - I dont think any other EU countries tax stocks in this way?

I thought that were more, but a quick search only revealed Norway, apart from Spain that you've already mentioned.

Also worth noting that in the Madrid region they don't have the wealth tax.

Also Denmark I believe

Especially given that many people will elect to be stably and cheaply employed and do just as good work as someone asking a “royalties model” fee basis for this kind of solution, it’s hard to see the advantage of going it alone. The competition is the ”cheap” FTE.

This is pretty much the only option for a person to have any sort of ownership... at least in the US where co-ops and the like are extremely rare w/ little legal help.

Honestly this guy probably got paid so much that the reward will be higher than many "successful" startup outcomes.

The biggest YouTube tail latency is “skip after x seconds”. And I don’t think this paper helps.

Reply on news.ycombinator.com