The end of Moore's Law forced YouTube to make its own video chip | Hacker News Reader

The end of Moore's Law forced YouTube to make its own video chip

techspot.com

111 pointsredbell54 comments

We talk power, not servers, at work because it’s the limiting resource at any scale. “How many MWatts do you need next year”. Its not that you can’t do these workloads on CPUs or GPUs - you can. But you can do it at 10x-30x less power on custom silicon.

That’s why Amazon did Graviton3; Google did TPUs and VPUs. Ditto on personal computer with Apple & M1/M2.

You skipped over the entire smart phone era.

Sure - add the Apple A chips; Qualcomm chips for communications, etc…

An important caveat is that this is practically and theoretically true for "embarrassingly parallel" (in the term-of-art, not perjorative sense) compute problems, and only if one sets aside supply-chain issues.

It's theoretically (but rarely practically) true for network-bound problems of all sorts.

It's not theoretically (and usually not practically) true for storage-bound problems. (Though Wh/bytes-at-rest is a useful bounding metric to understand if you're trying to build a storage business. It's never zero if the integrity of a byte at rest is at all important. Integrity is complicated.)

But in any case, supply chain issues tend to dominate as the limiting factor at the largest scale in my experience. This has always been true, even since before the current clusterfuck.

Source: no comment.

It's not obvious from this article why they need their own original chip. Video transcoding is a quite common task done by dedicated chip (or circuit, like QSV).

Perhaps because they push new codecs VP9 and AV1, these are also general format but Google is a developer and early adapter. Also perhaps their use case (batch processing, transmitted many times) isn't very suitable for existing transcoding chips that might be optimized for realtime transcoding.

The actual papers are very detailed.

https://ieeexplore.ieee.org/document/9567040

https://dl.acm.org/doi/abs/10.1145/3445814.3446723

They basically have computing per energy as an efficiency spec and then they show how that spec increased by 20-33x compared to an "Intel Skylake" machine.

Google also added some features, like a custom speed vs quality tuning thing, and single and multi-output transcoding.

What was interesting to me is the video core was made with "Catapult, a C++ HLS flow from Siemens".

Compute per watt is a great way to look at it.

A very trivial example of this is the concept in homelab circles to compare a cpus geekbench score to its watts to its price. Cost per point.

That is interesting. I guess it makes the most sense if you already have a full C++ implementation, which they do.

There's also a recording of the conference presentation at https://www.youtube.com/watch?v=vdXCVuY2CJs.

Siemens bought several EDA companies over the last ten years or so, in case anyone wonders why that old German company is offering advanced software.

I think it's the batch thing. They can put in one input file and get out multiple formats and resolutions. Also, they have very specific requirements like only encoding 5 seconds at a time for DASH streaming, so they can build the chips with a specific amount of RAM and leave out features like B-frames or whatever they're not using.

Typically one chip has the hardware for a total encode throughput of like 4k60 pixels per second, optimized for the possibility of one real-time 4k60 encode. The chip also includes an entire GPU and/or a bunch of CPU cores. Putting more fixed encode HW instead of GPU or CPU cores is more useful and efficient for YouTube, and at the same time they don’t need individual encodes to be real-time, so it becomes possible to design the hardware for higher quality if you don’t need to guarantee 16ms frame encode times (maybe? at least, "realtime" is the reason I got when asked why hardware can't use recon for intra mode decision...)

Intel went partly this way by slapping three 4-core Xeons on a PCIe card (VCA2), abandoned it, then announced variants of Xe to replace it. But ultimately they’re repurposing their existing silicon and it’s not as important for Intel as it is for Youtube.

The article [1] is a bit old (2010) but still relevant to answer your question. It compares the efficiency of a HD.264 implemented first on a general purpose CPU, then on a SIMD+VLIW DSP, then on the same DSP with additional optimized instruction, then on this platform with macro memory based accelerators, and lastly a pure ASIC implementation. There are significant efficiency gains at each steps, the big gap is between optimized ops and macro accelerators.

Flexibility do have a price. With efficiency gains due to new nodes getting lower (and pricer!), it's normal to see more dedicated hardware accelerators.

[1] https://courses.cs.washington.edu/courses/cse591n/10au/Paper...

not every hardware-encoding AVC (H264) or HEVC (H265) chip is in any way comparable once you start digging down into the various tweaks of encoder settings that are possible. it's very likely that youtube wants capabilities that they may not be able to guarantee exist in commercially available H264/H265 and similar encoding ASICs.

You can get similar functionality very cheaply on AWS from the vt1 instance type. They're a PITA to work with, which is probably why they're still around $160/mo. for enough transcode to power a small city on spot. These cards can transcode a full ladder (5 resolutions) of a 60 minute 1080p video in around 4 minutes, or the full ladder for 8 60 minute videos in parallel in about 30 minutes. Those instances are based on off-the-shelf cards that cost somewhere around $2000 each.

Guessing Google went the custom route for wider codec support, but standard hardware for this stuff is readily available in the professional broadcast industry

Would be funny if Google did custom hardware entirely for VP8/9, since VT1 on the face of it doesn't seem to support that. But more likely, Google didn't want to rely on a competitor, and in general with datacenters that have a reputation for doing things their own more-optimized way even if it's a hassle.

On an unrelated note, I find it funny that we call things like “Moore’s law” laws at all. Everything in science is a theory now, and these strictly observational “laws” are borne on much shakier grounds.

Meanwhile, everybody expects the law to break at some point (relatively soon, if not already). So what is it then? The observation formerly known as Moore’s law? Seems a bit of a silly way to chronicle contemporary, short lived phenomena.

Got a better name for those waves of temporarily useful theory?

Meme? Moore’s meme.

> I find it funny that we call things like “Moore’s law” laws at all

Because “Moore’s contemporary short lived phenomena” isn’t as catchy? Words can be used in all sorts of non-literal ways.

Moore conjecture? Moore approximation? Moore prediction? Moore relation?

Perhaps the Moore curve?

It's a law, not every law needs to be true / can be refuted or superseded.

So it'd still be Moore 's law, just not true and relevant anymore.

> I find it funny that we call things like “Moore’s law” laws at all.

I think it had more to do with 'Moore' rhyming with the word 'law'.

There hasn't been a radical shift in the use of the word 'law' over time.

wouldn't Moore's Lore rhyme a bit better

I used to think similarly about various "laws" but if you think about it for a moment, laws in what I suspect is their oldest usage refer to governance from a society laws. Those laws aren't immutable even though we pretend they are, they flex quite often. In some countries like the US they're designed to be inherently mutable. Laws can change, drop, be invalidated and so on. So it's not too inaccurate to call fairly solid theories that seem to behave in a stable manner as a law because laws are by no means absolute, either.

It was clear not long after Moore's law that it likely wouldn't continue indefinitely unless we could work at the quantum scale somehow, there was no path for scaling beyond that. We could extend that to Planck scale if we wanted to reach absolute limits of understanding but you can only double density so many times unless we don't understand space and matter that well.

However, speaking of governance, what governs Moore’s “law”?

Unlike the “laws of nature” or “laws of society” there is no such clear expectation of a governance in this case. Moore’s law is a rate observation. If it’s governed by anything, it represents our global investment in technological advancement.

Well with laws in society we do know sort of axomatically where they come from because they're human defined and we can reference the base. So I can't argue there too much, although there's plenty of squishy area around the edges where law isn't clearly defined but cases are attempted to be interpreted which boils down to some set of conclusions based on some argumentative evidence.

When it comes to laws of nature I would argue there's no clear governance in the laws of nature either. Why is the speed of light casualty? Why does space and time behave according to general relativity? Heck, I don't even know why my software behaves the way it does sometimes and it is well defined but it's so complex and behaviors become emergent that even that struggles to find a definite cause.

Laws are often just observational trends, pretty much always in science. Science itself isn't definitive and is structured under that assumption of change. All laws are is a culmination of defining a pattern our best understanding of observations and prodding at the universe. What governs the law of conservation of mass (hint it was taught as a "law" when I was in school but clearly isn't after nuclear physics and GR discoveries). Even Newton's "laws" are flawed as we know now and were just very amazing pattern approximations that apply to most but not all cases--what governs these are really GR/SR, but we don't know what governs them. Take black holes at the event horizon and deeper--we really don't know what's going on, we're just taking our well defined and studied sets of known patterns and trying to imagine and extrapolate what goes on. I wouldn't fault Moore's law too much, it may lack some of the deep rigor we expect in smaller sgstems but it was a pretty good and useful observation for quite some time.

We need to shed the idea that science will always give us the deepest underlying why, because philosophically it may not. It may get us very close and is a useful endeavor but we may never understand the causes of the most core relationships we discover. We like to think of laws like axioms in mathematics but they simply aren't in pretty much every case I can think of.

I submitted [1] a similar story when they announced it a year ago.

Google VCU Video Coding Unit at Hot Chips 33. It is also sort of strange ( or interesting ) their PR hype on AV1 has toned down a lot. To the point of practically silent.

[1] https://news.ycombinator.com/item?id=28302975

Moore's law is not over. It's not about density. It's about quantity of transistors on a chip.

It depends how literally one wants to take the law.

The "law" is doubling density every two years.

So strictly speaking it is done because it has taken longer than two years. But if you choose the looser interpretation and simple read it as, transistor density increases with time, then it's not dead.

Probably the bigger deal is cost per transistor is actually going up with the new nodes.

https://www.fabricatedknowledge.com/p/the-rising-tide-of-sem...

and

https://www.eetimes.com/moores-law-dead-by-2022-expert-says/

Koomey’s Law is where it's at - performance per watt growth is slowing - but it's still doubling every ~2.6 years.

People often misinterpret this because they think Moore was only talking about technology. However the number of transistors is more interesting than the technology alone because it represents what people actually do with chips. e.g. GPUs added to CPUs, and later AI added to the M1. These additions may not have been much more dense technologically, but the needs and desires of users increase as well as the technology. The tech doesn't have to improve 2x to reach 2x, because peoples' needs for AI, games, etc. is also increasing. And that's more useful in anticipating and predicting markets than the mere tech process.

So if you still think you're right, kindly post your citations. Here are mine:

"Moore's law is a term used to refer to the observation made by Gordon Moore in 1965 that the number of transistors in a dense integrated circuit (IC) doubles about every two years. "[0]

"Moore's law is the observation that the number of transistors in a dense integrated circuit (IC) doubles about every two years"[1]

"Moore's Law refers to Gordon Moore's perception that the number of transistors on a microchip doubles every two years..."[2]

[0] https://www.synopsys.com/glossary/what-is-moores-law.html

[1] https://en.wikipedia.org/wiki/Moore%27s_law

[2] https://www.investopedia.com/terms/m/mooreslaw.asp

Perhaps the bigger deal here is the loss of Dennard scaling in the 2000s. This is due to heat density at any given point on the chip. After all, increased transistor density could lead you to using all those transistors, but the thermal conductivity of all the materials involved isn't changing much, so there's a natural limit to the electrically active transistor density. You're welcome to increase transistor density, so long as you keep the number of active transistors roughly constant (not quite true since smaller transistors produce less heat).

You could keep transistor density fixed, but still double the number of transistors in a chip ... by using bigger chips. And that is naturally more expensive because they are more likely to be defective and require more sophisticated packaging. 3D stacking of smaller chips is another approach, since we usually only talk about the 2D area of a chip. This requires more advanced packaging, but substantially increases the likelihood of building a functional device.

The loss of Dennard scaling makes it more interesting to investigate ASICs because CPUs won't necessarily eat your lunch in two years. And circuit cleverness is one contributor to Moore's Law that Moore talked about in his 1975 address.

So Moore's Law is in some ways dead (number of instantaneously useful transistors is NOT doubling every two years) and other ways alive (useful work is improving rapidly so long as you can decompose your workload into leaning on the accelerators like YouTube did), depending on how you measure it.

Everyone downvoting, please post your disputatious citations.

Because even if you measure it by doubling the transistor and ignoring die size, aka density, the straight line graph is still broke.

The original paper clearly states one is supposed to keep the size of the piece of semiconductor and package constant. This is important for the cost argument which is featured predominately in the paper and is really the heart of the thing.

https://hasler.ece.gatech.edu/Published_papers/Technology_ov...

Even for transistor density, it isn’t quite true. The transistors aren’t getting any smaller like they use to. A lot of growth now involves stacking transistors on top of each other. 3D transistors lets you pack more of them closer together. While this allows for further increases in transistor density, you can argue that it is no longer an exponential function anymore.

> A lot of growth now involves stacking transistors on top of each other.

Approximately none of the growth thus far involves stacking transistors on top of each other, unless you count 3D NAND flash memory. FinFETs and upcoming GAAFETs are less planar than traditional transistor designs, but still don't get you two or more separate transistors stacked on top of each other. Actual 3D stacking of transistors on a single wafer is still nothing more than a hot topic for R&D, and stacking multiple dies full of transistors is pretty much only done with memory so far.

Well I was counting on NAND flash. That's how are getting more transistors per chip with them. The memory cells have basically stopped shrinking in NAND and actually got bigger when we first started to stack them.

FinFETs effectively lets us pack transistors closer. This is because much of the structure of a transistor is in a 3D fin that sticks out of the wafer. By making the fin taller, you can pack the transistors even closer. But it can be argued that by volume, transistors are basically the same size. GAAFETs lets us go even further with this idea.

Moore's law is definitely over. Chips are still getting faster, but that's not the same thing.

It doesn't seem dead quite yet.

https://static1.makeuseofimages.com/wordpress/wp-content/upl...

This is the number of transistors per device only. There is no notion of "minimum cost point" as in Moore's 1965 paper, where the "law" comes from. See my other post in this thread on this and why the economic part of Moore's law is as important as the tech part (density).

But that these chips don’t cost a million dollars shows that the cost per transistor has plunged commensurately, no?

It shows that the difference in efficiency of doing a custom ASIC at Youtube scale saves more money that what's required to design and manufacture their custom chip. But this is different from the trend cost per transistor (particularly for smaller players!).

And actually, this is the case because new processes efficiency gains are lower than in the past. It's something that's been said before (and I read about it on HN, it was just years back and I don't have the reference at hand now ;), but when process improvements were fast making a custom ASIC didn't really make sense: by the time the ASIC was designed and debugged and deployed, a few years were spent and a recent CPU was nearly as good as the custom ASIC for no risk and less effort.

Doing an ASIC is hard and takes time, and having a slower pace of improvement for new nodes efficiency makes it easier to justify the long deployment. There is now time to use such an ASIC with a noticeable gain, and absorb the cost and gain. Still, this is for a Youtube or Google (TPU) or Amazon (Graviton) or Apple. You need this scale still to absorb the development cost.

Are you going by vibes, rumors, Chinese whispers, or facts?

> definitely

Would you care to post some definite citations? Or would you like to dispute my citations listed above?

Moore's 1965 paper was literally about "the complexity for minimum component costs" [1]. So you're right that transistors on a chip is more applicable then just die density. But as too often nowadays you left out the "minimum cost" aspect. Moore's law is not only about tech performance, it's about economics. Everybody knew the "minimum cost" part would fall the first, and it did.

When the least cost offering improves, when things get better and cheaper, everyone follows. The lowering cost per function increases the addressable market, which can then sustain increasing investments. This is the virtuous cycle that moved the industry forward for a long time.

When the economic part stops, when you can get more transistors but only at a higher cost, then the virtuous cycle stops. Technical progress do continue, but you will have less players at each new step: only those who can absorb the increasing costs will keep on playing at the leading edge. And production costs will still rise. If the market gets smaller due to increasing prices, and the production cost keep on raising, you will see a slow down as it will take more time to amortize investments.

The big players are worried about this: slow down has in itself the specter of followers catching up (eventually...), leading to commoditization. We're definitely not there yet, but it's not a good perspective. Definitely bad for their stock. So the cost has been taken out. At first the NRE part was dropped of the equation, to consider only the cost per transistor. For this one can also be creative: considering the fabs amortization, or after? Makes a big difference (see the issue car makers have in getting new investments at "old" nodes: message from the big guys is move to smaller nodes, even if it's not always convenient). But in the end it's been simpler to just drop the least cost aspect completely. The industry PR had good results on this ;)

For small to medium fabless players, Moore's law has been over for a while. For the biggest players it's also over now: there's still technical progress on transistors per device, but at an increasing cost.

It's not always a smooth process: if you're an Apple, you can gain on the cost by going vertical (good-bye Intel margin). But it's a bold move not for all, and it's a one off gain.

Moore's law is over. Tech progress is not. For HN readers who can afford the increasing cost of the devices and are reading regularly from the big silicon vendors that "Moore's law is still going strong yoo-hoo", when it's been neutered to transistors count only, it may be easy to miss.

[1] https://newsroom.intel.com/wp-content/uploads/sites/11/2018/...

Does it annoy anyone else that many articles insist on quoting part of the article in a larger font to highlight it? It's a relic left over from magazines where it would draw the reader's attention to a particular segment.

It makes no sense whatsoever to do it on a web article that you scroll through to read. This particular one contains the text "After a 10-minute meeting with YouTube chief Susan Wojcicki, the company's first video chip project was approved." then it's repeated immediately after in a larger font.

Maybe we'll need co-processors in the future again?

Although when you think about it, TPUs / VPUs etc. being pluggable to a PCIe slot is mostly getting us there?

We've already got them! They're integrated into the "System on Chip" or as chiplets on the package. Just considering Apple's M1: https://en.wikipedia.org/wiki/Apple_M1

Because pure-software implementations of algorithms are too "expensive" in terms of power and/or latency, the M1 package has logic blocks that implements a GPU, a Neural Network processor, and H.264/H.265/VP9/JPEG/ProRes codecs. When Apple quotes battery life, the fine print says "1080P HD video with brightness set 8 clicks from bottom". Powering down most of the cores and lighting up just one of the codecs is clearly a good way to achieve 18 hours of battery life.

AMD and Nvidia sell GPU "co-processors" that plays games and has been used for machine learning in "GPU mode" (think games using "software rendering"), but Nvidia GPUs now have "Tensor Cores" to accelerate and decrease the electrical power required for some of the linear algebra functions common with ML. Laptop chips from Intel and AMD will have embedded video on package to eliminate an external GPU.

...or they needed a chip that supported av1 and despite all the major tech companies being a part of the working group for years none of them had implemented it because Nvidia has a stranglehold on the industry because it makes money from h26x codecs

It must be a law of common sense that if all you do is encoding every stupid video on planet, you will need an encoding accelerator because gpu manufacturers likes to lock down and charge alot for tiny part of silicon that does it.

> because gpu manufacturers likes to lock down and charge alot for tiny part of silicon that does it.

Those shenanigans have a far smaller effect than the fundamental problem with buying a large GPU when you have no use for 80% of what's on that chip. That's obviously not going to be economical at large scale (especially not during a GPU shortage driven by demand for exactly the portions of a GPU that you never use).

Yt milking tf out of ‘that one time they designed an ASIC’

Reply on news.ycombinator.com