Hacker News Reader

Watch the presentation from 6 months ago, where they explain the decision to build their own hardware for inferring : https://youtu.be/Ucp0TTmvqOE?t=4309

It's not surprising that they also build the hardware for training. Correct me if I'm wrong, but Google use the same TPUs for training and inference, because the underlying operations are the same : multiply then add numbers. Once Tesla built the hardware for inferring, the design of the hardware for training is probably similar.

Unlike Google's TPUs, Tesla have a specific use case for the hardware (computer vision for automotive), and maybe than means they can further optimize the computation pipeline with their own specialized hardware.

Google has Edge TPUs for use outside datacenters, and they don't support training. Neither do the chips Tesla made for their cars. It's a pretty different problem.

I wouldn't be so sure. Edge TPUs could be the exact same architecture than Google Cloud TPUs, but as you need less computation power for inferring than training, they have simply less transistors on the die and could be underclocked.

In other words, Cloud TPUs could be the same architecture than Edge TPUs but scaled to an higher frequency and more packed.

I guess we need sources to confirm.

Edge TPUs support 8 bit integer math only. Training is floating point. That's not a small change. https://coral.withgoogle.com/docs/edgetpu/models-intro/#mode...

Training is currently done in floating point math, whereas inference can be done fixed point without much loss of performance. Fixed point is ~10x cheaper in terms of power and silicon area for equal performance.

Also, training requires a lot more RAM per unit of compute, since it needs to store all past layer activations, whereas for inference, that is unnecessary.

As far as I know, no player who has developed dedicated ML hardware (as opposed to using GPU's) uses the same hardware for both inference and training.

Very good video, it contains answers to many of the questions that people are speculating here about and other interesting things about Tesla's custom chip.

- It's under 100W so they can retrofit into old cars

- lower part cost, so they can do full redundancy with doubling the parts

- they estimated that 50 TOPS is needed for self-driving

- lower latency with batch size of 1 compared to TPU's 256

+ GPU for post-processing

- security: only code signed by Tesla can run on the chip

- at the time (2016) there was no neural net accelerator chips

- some part's are built from bought IP (so not reinventing them) Probably things like the 12 ARM CPUs, LP DDR4 memory, video encoder, maybe the separate post-processing GPU too...

- physical size of the board is small

- performance example: on CPU 1.5 FPS, on GPU (600 gflop) 17 FPS, on Tesla's NN accelerator 2100 FPS

- Besides the convolution even the ReLU and Pooling is implemented in hardware

- Paying attention to the energy efficiency down to the arithmetic and data type usage.

- The silicon cost is less than their previous hardware (HW 2.5)

- old hardware 110 FPS new one 2300 FPS

- 144 TOPS compared to NVidia's Drive Xavier 21 TOPS

I think the size of the networks they are training might already be good motivation for developing custom hardware for training.

I would expect their training hardware to be something specifically aimed at optimizing memory bandwidth to support distributing training of their “shared” hydra feature. It’s interesting that the shared hydra feature extractor is able to converge as they keep adding more and more output predictions under a training regime of interleaving asynchronous updates to the model from different predictor networks ...

Seems to me the formula they are pursuing with custom hardware might be to support a strategy of 1. keep adding more predictions based on same feature 2. Increase the span of time represented by batches used to train the recurrent networks

Both pursuits seem very data efficient in terms of the amount of training data they could conceivably collect per unit time of observation ...

Custom hardware with a problem specific memory architecture aimed at efficiently supporting training with very large rnn time slices could be developed that’s more about “make it possible to train this proposed model at all” rather than “make it faster/cheaper to train existing common model architectures”. When custom hardware is required to make it possible to train the model they want, the validity of the hardware development cost bet might end up being more about the effectiveness of the model they think they want than it is about maintaining general purpose performance parity vs any off the shelf hardware options ...

At Tesla's scale and priorities, they'd probably be less keen on using external cloud providers. Using TPUs at their scale would certainly require Google's AI consultants to supervise which isn't ideal for Tesla.

Not agreeing or disagreeing with their decisions, but if you have the resources, you can certainly design a custom chip that performs a specific type of task very well that beats other competitors. Nvidia's GPUs are have to be reasonably good at training across different NNs. You could have a chip that's exceptional good at training one/two specific types of tasks.

For most companies, this would be a bad idea. However, Tesla knows how to produce hardware.

> At Tesla's scale and priorities, they'd probably be less keen on using external cloud providers.

By hardware-hours, Tesla is hardly one of the top companies training deep networks. Planet Labs (satellite imaging), Netflix, Pornhub, to name a few.

What's the info they'd leak to the Google consultants? How much data or TPUs they're using? This is practically public information.

What does Netflix do with NNs?

Their own page says, "our recommendation algorithms ... learning characteristics that make content successful ... optimize the production of original movies and TV shows ... optimize video and audio encoding, adaptive bitrate selection, and our in-house Content Delivery Network ... and advertising".

https://research.netflix.com/research-area/machine-learning

Here's another article on the subject:

https://becominghuman.ai/how-netflix-uses-ai-and-machine-lea...

At a first glance, they don't seem to be problems in which ML learning can have a huge impact. Netflix in the US has a catalogue of about 4000 movies plus a few hundred tv series. It's tiny- compare with 153707 items on a niche recommendation website (criticker.com).

Characteristics that make content successful.. here again, it's mostly decent quality content plus marketing. I doubt the scripts are reviewed by NNs.

I have no idea about the network and delivery stuff, but I guess a well designed network can take care of most of it.

My strong impression is that, similar to other well known cases (Uber, WeWork) Netflix is a mostly traditional company (a media company) that very strongly wants to be seen as a tech company.

Netflix put a lot of work into recommendation back in the dvd delivery days when their catalog was absolutely massive. Now in the streaming space the catalog they license is much smaller so recommendation is less important; basically they just advertise the popular stuff for your demographic.

Its a bit odd that legally one can rent out physical disks, but there is no corresponding way to legally get permission to rent out streaming content without negotiating with the rightsholder. But thats how it is...

> Its a bit odd that legally one can rent out physical disks, but there is no corresponding way to legally get permission to rent out streaming content without negotiating with the rightsholder. But thats how it is...

It's hard to think of a good fair regime to do this under.

At least with a physical disk, there's a maximum reasonable rate that you can turn the disk around between users and you need to have enough copies for whatever the lifecycle peak demand is.

We do have an audio compulsory licensing system for things that are purely songs. But with video works, there's not a clear boundary for "how big" the work is-- how do you treat 30 hour anime series vs a 5 minute Pixar short? How do you treat continuing medical education videos vs. fluff amateur made content? Etc.

Do those really take more than 70,000 gpu hours to train a model?

> At Tesla's scale and priorities, they'd probably be less keen on using external cloud providers.

Not sure if it’s still the case today, but previously Tesla’s training was done on-prem and with their own in-house Tensorflow clone.

And yes, if you get TPUs from GCloud, you are likely to be working with their engineers to get things working. Those engineers tend not to have much business conflict of interest, though. They want to help you because your problems are likely more interesting than what they’d otherwise be assigned.

Nothing crazy about it. TPU-like stuff is ~10x the energy efficiency of GPUs and several times the speed. When you're spending megawatt-hours and days to train a single model, it adds up in both real and opportunity costs.

Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google (and I'm assuming with other companies under Alphabet umbrella), at Google's sole determination. Not that it would be a good idea to upload Tesla's proprietary data into Google Cloud even if it did not. Cloud, after all, is just somebody else's computer.

Stuff like their TPU and Waymo's Honeycomb Laserbear (something along those lines... their lidar naming system is pretty long) shows that Google is making good products for a limited reach of people.

TPU? Seems like it has a lot of potential, but not for people directly competing with them.

Waymo's Laserbear lidar? Seems like it has a lot of potential, but not for AV companies directly competing with them.

Google's playing this game pretty fiercely... which given their size is pretty bad/daunting.

> Google TPU TOS prohibits the use of TPUs for stuff that competes with Google

I don't think this is true. If you're talking about https://news.ycombinator.com/item?id=19855099, it doesn't apply to TPU hardware, as is explained in the comments there.

> TPU-like stuff is ~10x the energy efficiency of GPUs

10x is probably overstating it when talking about newer GPUs because they have ML hardware in them now. Also, that still doesn't make it a good idea to build your own chips because there will soon be many third party options to choose from. Doing your own chips is a bet that you will out execute dozens of companies ranging from startups to industry giants. Simply taking your pick of the best commercially available options is likely to be a better choice in the near future.

Yep that's the clause. The clause itself is not that problematic for Tesla. What's problematic is that it can be changed over time, and it'd be foolish to single-source something as important as deep learning compute without the option to go elsewhere. Not to mention the rather extravagant Cloud pricing. So Tesla is taking a page out of Steve Jobs' playbook and it will control its own core tech. That's smart, especially considering that they already have bits and pieces of the IP that they'll need.

That clause doesn't apply at all though. It's not even for the same product.

As for single-source, the models are written in PyTorch, not TPU machine code, and they're pretty standard models anyway (e.g. Resnet-50). They can easily transfer to other hardware if necessary. There's not a ton of lock-in there. It doesn't justify the massive costs of ASIC development just to avoid this nonexistent lock-in and imaginary TOS clause.

> Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google

Can it be true? Then again, Apple's app store behaviour seems to suggest such demands are tolerated. Antitrust is really asleep in the US, isn't it.

>Nothing crazy about it. TPU-like stuff is ~10x the energy efficiency of GPUs and several times the speed. When you're spending megawatt-hours and days to train a single model, it adds up in both real and opportunity costs.

could you share the stats on this? Google told me to use a K-80 for training.

Not remotely true. TPUs and GPUs are neck in neck with each other right now w/r to overall efficiency, check out https://mlperf.org/press#mlperf-training-v0.6-results for more details.

GPU advantage: more refined ecosystem and you can buy them for $<1000 or get laptops with them built in, and if NVDA has sweat more software engineering blood and tears than GOOG into your model's functions, it will run better on them

TPU advantage: Colab has a free tier that lets you play with them at no charge and if GOOG has sweat more software engineering blood and tears into your model's functions, it will run better on them.

All IMO of course. And deep down it can get more complicated than that, but I salute GOOG for being the first company to ship competitive AI HW, doubly so at scale.

Also, the software part of it (NNs and their algorithms) have been so widely researched and published that competitive advantages here are harder to come by than in hardware RD.

Also, vendor lock-in is a huge challenge in the cloud space. I don’t think Tesla would be comfortable with the fact that all their training data sits on a potential competitor’s datacenter.

A car is a hardware device as well, and an electric car does not have the kind of power budget that allows you to throw oodles of standard pieces at it without paying a severe penalty in range.

Compared to the energy needed to move the car, everything else is pretty irrelevant. Power hungry features like Heating/AC only makes a few % difference to range.

From autonomy day hacker news comment: "Pegasus consumes about 500Watts, compared to under 100 Watts for Tesla's FSD computer. Elon in particular emphasized the performance per watt (as it's always possible to cram more chips to increase performance if you ignore cost and power consumption). The comparison made in the video: 500Watts for an hour consumes about 2-3 miles of range. In a city in slow traffic, going 12mph, that's a significant range reduction. So you might have a 10% improvement in range for the Tesla ASIC in low speed conditions" [0].

Also, Tesla is/was planning on running these chips even while not in autonomous mode in order to spot new scenarios which it can then record what the car sees and what the human does in order to collect a lot more unique road scenarios to train their models with.

[0] Credit Robotbeat: https://news.ycombinator.com/item?id=19729743

Good ol' velocity-cubed.

Increasing their compute per watt allows them to get more compute per watt, so they will pursue it lots, low speed energy savings are a way of making the range extension seem like a big deal when it isn't (maintaining highway speeds requires tens of kilowatts).

It was a big deal for Tesla, though, because they had to fit into a tiny power budget. HW2.0 wasn't enough for what they wanted to do, and to retrofit existing cars they had to consume a similar amount of power.

Power efficiency does make a range difference and is worth seeking, but Tesla is exaggerating this, IMO, to conceal one of the ways that the churn-heavy cycle for FSD has imposed organizational costs (an entire chip-level hardware program, in this case).

I would expect them to seek compute/watt basically until they stopped making progress.

An actual working system might slow them down, but even then, better hardware is one of the ways to make it better.

Designing ICs is a very expensive treadmill. If you can buy fast enough hardware to do your job, it's really hard to justify spending a lot for something that may be nominally a little better on some metric.

There's commodity computing of comparable throughput that other automakers can incorporate without too much cost, but it doesn't fit into the existing TM3 vehicles because it has a larger footprint in power, space, and cooling than HW2.

Here, Tesla spends a bunch of money to develop HW3 and overcome this problem, but the advantage in a new car design between HW3's efficiency and commodity hardware is limited.

Wow, I stand corrected

The compute for autonomous driving could be 10% of the total energy budget of the car. https://cleantechnica.com/2017/10/13/autonomous-cars-shorter...

So would you trust the company that owns one of your biggest competitors in this field (Waymo) with the stuff that decides over success: data?

Did they say they were building their own training hardware? I thought it was just their inference hardware (the boards on the teslas)?

Yes https://youtu.be/oBklltKXtDE?t=572

Oops, didn't catch that. For some reason I thought it was a orchestration engine like Stripe's Railyard.

>Also, I'm really curious whether the custom hardware in the cars is benefiting them at all yet. Every feature they've released so far works fine on the previous generation hardware with 1/10 the compute power.

The latest OTA finally brings a hardware v3 only feature, traffic cone visualization, and traffic cone automatic lane change.

Traffic cones today,1 million self driving robo taxis in 6 months, right?

I would guess that while the new hardware has the same features, the accuracy might be lower on the old GPU's because they are forced to use smaller networks or to run them at lower frame rates.

Stay ahead? Are they actually ahead?