""" One may ask whether the conditionality introduced by the use of CMM does not make FFFs incompatible with the processes and hardware already in place for dense matrix multiplication and deep learning more broadly. In short, the answer is “No, it does not, save for some increased caching complexity." """
It's hard to beat the hardware lottery!
> We therefore leave the attention layers untouched
Meaning, presumably, that the GPU memory remains the bottleneck
Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!
I'm really interested, can you share where you got these numbers?
8-bit ops, inference only, low memory embedded, excluding the host, implied utilization from FPS specs is ~20%
But the trend is there.
There are also newer ADAS/AV units from China which claim 1000tflops and cant really cost more than $1000/$2000 per car.
These are all tiled designed (see also dojo/tesla) heavily over-weighed on flops vs memory
H100 is basically ~$2/(2000 tflops/s)/hour or $1 for 4*10^18 floating point operations.