Hacker News Reader

Comment by rvnx | Hacker News Reader

I think nobody said Netflix' infrastructure can be built in a weekend. However, the scale doesn't matter that much after a certain point once the scaling "wall" has been pierced. If you are a biscuit factory that produces 100'000'000 biscuits per year or 500'000'000 biscuit per year then the gap between 100M and 500M isn't that impressive so much anymore as it's mostly about scaling existing processes. However, if you turn a 1'000 biscuit shop into a 1'000'000 biscuits company then it's very impressive.

View on HN

Nonsense.

It's still impressive. A 5x increase at that scale can be a phenomenal challenge. Where do you source the ingredients? Where do you build the factories (plural because at that scale you almost certainly have multiple locations in different geographic locales subject to different regulatory structures). Where do you hire the people? How do you manage it? What about the storage and shipping and maintenance of all the equipment and on and on? How much do you do in house how much do you outsource to partners? What happens when a partner goes belly up or can't meet your ever increasing needs?

Your comment is a great example of what the OP pointed out.

My favourite example of these sort of extreme scaling issues are the fact that McDonald's apparently declined to sell products with blueberries in them because modelling showed they'd have to buy the world's entire supply of blueberries in order to do so.

I thought this was hyperbolic, so I looked into it:

> The menu team comes up with interesting ideas like including kale in salads. The procurement team and suppliers then try to get the menu team to understand the challenges. How do you bring kale to 14,000 restaurants? As one example, when they introduced Blueberry Smoothies in the U.S., McDonald’s ended up consuming one third of the blueberry market overnight.

https://www.forbes.com/sites/stevebanker/2015/10/14/mcdonald...

I couldn't find any other source to back it up, but still wow! That's an absurd number.

McDonald's sells blueberry muffins

So there is an extreme dearth of blueberries I guess compared to other food goods? I mean, McDs isn’t taking over the entire supply of potatoes or chickens for example correct?

I think the point is that the supply chains probably need upwards of years of time to adapt in some cases, you can’t just turn on a recipe that needs a full cup of blueberries per serving on Monday and expect there to be a spare million cups of blueberries to be lying around the supply chain on Tuesday.

In the case in animal product, there are almost certainly major operations worldwide that have been built and financed purely to serve McDonalds demand. They probably have to even build these out well before entering some markets.

They grow a lot of potatoes in the US. Last week I hauled a load of tater tots destined for McDonalds. I’ve hauled potato products for McDonalds quite often.

They raise a lot of chickens in the US. I’ve hauled chicken nuggets or chicken breasts for McDonald’s in the past quite often.

I can’t even tell you where they grow blueberries.

It's the exact opposite.

Taking the software example, you can easily scale from 1 to 100 users on your own machine. You can handle thousands by moving to a shared host. Using off-the-shelf web servers and load balancers will help you serve a million+. From there on you'll have to spend a lot more effort optimizing and fixing bottlenecks to get to tens, maybe hundreds of millions. What if you want to handle a billion users? Five billion? Ten billion? It always gets harder, not easier.

Pushing the established limits of a problem takes exponentially more effort than reusing existing solutions, even though the marginal improvement may be a lot smaller. Getting from 99.9% to 99.99% efficiency takes more effort than getting from 90% to 99%, which takes more effort than getting from 50% to 90%.

You never pierce the scaling wall. It only keeps getting higher.

If you can serve 1K users with 10 employees, you can probably serve 1M users with 10k employees.

Remember that global productivity usually does not scale with headcount!

Each employee adds some overhead, which requires more employees... which requires more employees.

Sounds like the rocket equation! Perhaps big companies are rocket science?

And you can birth one baby in 3 months by 3 women, right?

To add something useful as well besides snark, first of all, there are hard physical limits, which are sometimes well within context (you really shouldn’t try to outcompete light speed for example, relevant in some high-freq trading, infrastructure projects). Then you can try to increase headcount to any number, you won’t produce for example a better compiler. There are simply jobs that are more “serial” - the only way to win at those is to try to employ the very best of the field in a small team.

No, just 3 babies in 9 months.

You can deliver DVDs to netflix subscribers as well to achieve a much bigger throughput, but I doubt they would be as popular as they are right now :D

Sneakernet!

“Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.“ -Andrew Tannenbaum

That's too simplistic. What about the doctors and medical facilities and other supporting infrastructure? What about the baby food and medicine and clothing and supplies and what about the people to take care of the children? You think you can just keep throwing more women at a hospital having babies to infinity and not have any problems?

There's a limit on those things, but it might as well be infinity when you're trying to have 1 baby or 3 babies or 100 babies.

1M users and 10k employees is not in the range where you have crushingly impactful logistics.

That won't help your customer expecting their 'baby' after three month due to the increased mother-workforce ;)

But the goal wasn't better or faster. It was giving more customers the same service. You're talking about a completely different problem.

If you told McDonald's to double their number of McRibs produced next year that would be an incredible challenge to meet. They already sell enough that it affects the global pork market, it'd be insane for them to double their demand for pork. What about other supplies, would this result in a reduced burger demand? How can they ensure they can respond appropriately either way? They probably run near fridge/storage capacity, does increasing this mean they need to also increase storage at restaurants?

That's a 2X increase. Now do it again and a half for a 5x. Crazy to say there's a "scaling wall" that once you "pierce" it's easy to scale up. It's the opposite, McDonald's already knows how to supply and sell X McRibs a year, there's no company that's ever sold 5X those McRibs so they have to figure it out themselves.

There's an old rule of thumb that each order of magnitude increase (10x) brings a whole new set of challenges.

Anecdotally I experienced this when scaling my software product from 1 --> 10 --> 100's --> 1000's etc. of users.

Thats not to say 2x can't be a substantial challenge, as you pointed out. It gets harder (and IMO more fun) when you're at the bleeding edge of your industry.

> the gap between 100M and 500M isn't that impressive

This is absolutely not true. The closer you are to peak performance, the harder it is to scale, and the returns diminish heavily. At many major tech companies, there's a huge amount of effort into just 1% - 5% optimizations -- these efforts really require creative thinking and complex engineering (not just "scaling existing processes".) At the volumes these companies operate, even a 1% optimization is quite significant.

Aren't you contradicting yourself?

If you're on 100M users you're probably scaling vertically. So adding 5x more hardware shouldn't be a problem.

But when you're at 500M all of a sudden it makes sense to optimize further since the capital saved will be the same percentage(ish) but the money is worth peoples time all of a sudden.

I know that we don't care particularly about power savings in the DCs I've worked in, because they're relatively small. While bigtech will do all kinds of shenanigans to save a couple watts here and there, because it's worth it across your hundreds of thousands of servers.

Seeing scale issues as purely hardware bound is incredibly naive. Even in a case like streaming, if you’re pushing more bits through the wire, it’s likely the increase in usage causing the traffic increase affects the software systems you have in place to support your service start degrading and you need to rearchitect them. Very few problems at that scale can be solved by throwing more hardware at them.

In the real world, it's rare to find a non trivial business where operational costs to its problem size scales linearly. Let's take an example of search. If t he web size grows by 10x; this not only means # of your user grows by 10x but also # of web pages to be indexed, size of the ranking model, ratio of unstructured/dynamic content formats, faster pace of content updates, reduced informational density of overall web, size of user logs, advanced SEO, (those are just a tip of iceberg, there are so many other stuffs) etc etc... Every single dimension scales and stacks up. And you need to handle all of that within a limited latency budget where you cannot really make any compromises.

> So adding 5x more hardware shouldn't be a problem.

If only things were that simple. https://en.wikipedia.org/wiki/Amdahl%27s_law

Part of it depends on if "build it five more times, again" is a viable strategy.

Building five "Netflixes" with identical content is possible; the amount of content wouldn't change (it would decrease, the cynic says); you just need parallel copies of everything (servers, bandwidth, etc).

The fun would come in syncing usernames, etc through the system.

It's an entirely different class of problem compared to "acquire resource, convert it, sell it".

It's a good point, and I think it's an interesting comparison. Obviously improving by a factor of 1000 is better than improving by a factor of 5. But the absolute improvement is still 4 times larger. 400'000'000 extra biscuits is going to bring a lot more revenue than 999'000 biscuits

> However, the scale doesn't matter that much after a certain point once the scaling "wall" has been pierced.

Sorry, you gotta overhaul majority of your architecture and its components to scale by every 10x. It's not a single "scaling wall" to break through but it's more of a relentless stream of uphill battles. And this gets even more interesting when you reached to the point where there's no prior art for your problem, usually at hundreds of billions of users.

Actually, you're incorrect. Scale problems seem to have quanta. In your example you will have physical issues with ingredients, etc at some point. It might be storage, it might be because you've run out of water. It might be because you've run out of electricity.

Making 5 things and making 5,000 things is as different as making 50,000 things and 1m things. There are always cost constraints at each level, and each design can only go so far.

Reply on news.ycombinator.com