You can normally expect to get way more than half the 'outcome' from a neural net with half the ram/compute/time/power budget. So neural nets scale 'down' pretty well.
Apple has done a wonderful job of further locking their user into the golden cage they call a platform.
ML is one of the few applications that benefit from platform-specific optimizations, so if you need every ounce of performance, you have your choice of which walled garden you want to tether your application to. The "lock-in" comes from the specific capabilities of your special-purpose hardware, and for serious applications, you're already thinking hard about whether to design your entire implementation around Apple, NVidia, Google/TPU, or even Android devices. For big models, platform-specific needs influence every aspect of model design, including data/model sharding, quantization, training loops...
For non-scientific applications, it's usual practice to train your model in platform-agnostic ways using PyTorch or Tensorflow or whatever and then deploy it to devices in platform-specific ways, whether that's XLA, CoreML, Edge TPU, Android NNAPI, TensorflowJS, or hell, custom-written GLSL shaders or whatever.
We're just starting to see cross-platform frameworks that abstract model inference: TFLite, PyTorch Mobile, ONNX. To their credit, CoreML can act as a backend for any of these, so you don't even need to worry about your platform.
EDIT: I fell for NVIDIA's marketing. The dense FP16 performance is only half of 284.48, which is 142. Thanks to adgjlsfhk1 for the correction.
It turns out that even without the extra training iterations you often lose surprisingly little in terms of quality of output. In reality you can sparsify a lot more, but 2 out of 4 is so simple and easy to implement in hardware, more complex schemes are much harder to justify.
However, small matmuls (say, <2048 bytes in the K dimension) won't get anywhere near 2x performance.
> An important distinction is that the AMX:CPU ratio is not 1:1; not every core has its own AMX co-processor.
My understanding is there's only 1 of those per regular M1 CPU, maybe 4 on the largest one (Ultra).
So that's actually just about as power-efficient for fp32 as a 3090, which according to wikipedia is 35 Tflops in 350W. Supposedly AMX can do 2x rate for fp16 as opposed to the 3090's 4x rate, so maybe 2x less efficient than a 3090 for fp16.
Interestingly, fp64 hits 370 Gflops at 15W...
cblas_sgemm: 36 GFLOP/s
vDSP_mmul: 41 GFLOP/s
edit: that seems to be verified in the tlkh.dev blog post above. Interestingly, I ran the same code on my bargain-basement 2020 iphone SE, and got 259GFLOP/s! These apple devices are pretty mindblowing.
Yes. Aside from benchmarks, you can easily verify this by profiling an application with Instruments and then inspecting the disassembly.
However, it should be said that AMX does not scale linearly with the number of cores, but with the number of core clusters. So, on the M1 if you use Accelerate in two threads (rather than one), performance will barely improve, because the first thread can keep the AMX unit busy enough.
However, e.g. the M1 Pro and M1 Max have two performance core clusters with AMX units in them. So matrix multiplication doubles roughly two times compared to the M1. Similarly, the M1 Ultra has fours performance core clusters, so matrix multiplication performance is roughly twice that of the M1 Pro/Max and four times that of the M1.
Benchmarks:
AMX is an unstable ISA that changes between product generations. That's why it's not publicly documented.
Arm SME is the standardisation of the concept, but is not inmarket yet.
https://community.arm.com/arm-community-blogs/b/architecture...