And yes, all their competitors are making custom chips. Google is on TPU v7. absolutely nobody is going to get this right on the first try among their competitors - Google didn't.
There is a pretty big moat for Google: extreme amounts of video data on their existing services and absolutely no dependence on Nvidia and it's 90% margin.
Its clear from listening to podcasts/interviews, he does not want to say anything to get on elons bad side. Interviewers appear to also not be eager to broach the subject.
That is true, but that doesn't mean Nvidia is not engaging in engineering to intentionally kneecap competition. Triton and other languages like that are a huge threat and CUtile is a means to combat that threat and prevent a hardware abstraction layer.
Those control flow ops aren't even supported on many backends. I know tensor rt doesn't support them for example, at least today.
Removing control flow isn't as easy as you'd think for some. It essentially means ripping large sections out of python and into separately implemented c++.
I am so confused by metas ecosystem. Perhaps others have the same issues. I have mountains of torchscript code. It worked fine for me - had no issues making the python compatible. Torchscript is now deprecated, and the ostensible replacement is torch.export and either: AOTInductor or executorch. torch.export is so limited - no control flow at runtime at all, less support of python than torchscript. It is far more work to hoist all the control flow out of the model than it ever was to make the model torchscript compatible. Feel like meta has moved on, but I'm still stuck in the past here.
I think my real problem is that xilinx pushes the gui flows heavily. It is extremely annoying to configure the mpsoc fabrics entirely outside of vivado. Same thing for using any of their bundled IP.
Does it actually creates a unified memory pool? it looks more like an accelerated backend for a collective communications library like nccl, which is very much not unified memory.
Might be helpful if they actually provided a programming model for ANE that isn't onnx. ANE not having a native development model just means software support will not be great.
why exactly do you need a deity to tell you to love your fellow man? Do you need god in your life to want to love your children? I think this is not quite right. I don't expect that the desire to create these tools independent of outcome in the valley is simply about greed , and for companies like anthropic, the ability to use AGI fear as a means to drive investment in themselves from VC class that lives the idea of obliterating human labor. We need less money in tech - we'll probably get it soon enough.
How does fex deal with the fact that the memory model on arm is weak and x86 is total store ordering. It seems like would need to hammer performance by putting memory barriers everywhere to handle all cases. Perhaps fex only works when there are well defined mutexes it can gain visibility into? anyone know?
yeah that is correct. The m series chips can turn on total store ordering memory model solely for Rosetta. There's also some hardware extensions to arm to support x86 condition codes in the hardware because it's way more instruction efficient that way.
The neural engine not having a native programming model makes it effectively a dead end for external model development. It seems like a legacy unit that was designed for cnns with limited receptive fields, and just isn't programmable enough to be useful for the total set of models and their operators available today.
MLX doesn't use the neural engine still right? I still wish they would abandon that unit and just center everything around metal and tensor units on the GPU.
I think the bigger deal for founders is that Nvidia can decide at a whim to deny you supply or, better yet give 100 billion dollars to your competitor.