...out of all the things I expected to see from a Tesla stream, I definitely did not expect a 25 minute discussion on the cost of 32bit additions, dot products, sram bandwidth, and chip design, at the level of a third year college hardware course.
this is wonderful
I don't know the context here but it feels like they're trying to ML-splain Karpathy?
"you say you can put bananas in your smoothie. just a thought, but perhaps, might I ask, do you have the capability to put strawberries as well, or are we not there yet?"
Basically (IIRC), during backprop the error difference gets ever smaller the further back in layers you go, ultimately getting "lost in the noise", making learning in the earlier layers more difficult to impossible.
I'm not saying RELU is the only option to make this work, or that it's the only activation function that provides a "fix" for the issue; I'm sure there are other ways to deal with vanishing gradients that I don't know about.
I also lack the mathematical knowledge as to why RELU helps in this manner, but I suspect something having to do with the lack of "asymptotic structure" approaching the extremes (I don't know what the proper term would be). Or maybe it allows for some form of "forgetting", in the prevention of multiplying very small numbers (such values just go to zero ultimately)?
Maybe someone else here with the knowledge can explain it better, and we can both learn...?
This is what everyone that is not Tesla or SpaceX are doing. And have been doing for a long time. If the CEO is not an engineer at heart, what are they?
I seriously doubt Elon Musk has more engineering knowledge than the people he hires on their specific fields. However, he can make pretty well informed strategic decisions if he knows WTF the engineers are talking about without taking their word – not even that, as explanations have to be dumbed down.
This is not a new thing. Bill Gates was like that (1). Steve Jobs was no dummy and had an engineering background, but not at the same level – he did parter with a genius engineer, however.
I think Musk is doing the right thing.
> Which might explain why he's a crummy CEO
That's quite debatable, I'd say. Isn't he getting results?
(1) https://www.joelonsoftware.com/2006/06/16/my-first-billg-rev...
I think it's very important to point out here that Elon does not run SpaceX. It's well known that he's a front man for that company, and that all day to day operations are run by an actual executive who does the job of being an executive. This is why SpaceX is doing much better than Tesla.
Sure, if you call burning billions of dollars a result.
If your goal is to make money, I'd follow anyone but Tesla or SpaceX's example.
Despite the fact that I own a Model 3 and own shares in Tesla I often find the critics make good points and just hope for the best anyways because I want electric cars to take over and I want self driving cars to develop sooner than later because driving is dangerous and tedious.
The SpaceX hate seems much less justified to me. SpaceX has been building launching, landing, and reusing rockets for years now. They've ushered in a new lower cost space age.