modern AI is just vector multipication. any AI chip is just 10,000s of very simple cores which can do vector float operations and little else. this also entails clever trade offs of shared cache and internal bandwidth.
(as a thought experiment, consider a naive million by million matrix multipication. this will take a single cpu about 1 year! how do we reduce this to 1s ?)
the end
I thought they were generally int8 or int16 vector multiply adds and occasionally float16 added in.
Each "unit of work" in matrix multiplication is not dependent on any other unit of work. Stuff as many cores as you can into a chip, and then simply feed in all your vectors at the same time.
I.e. basically a beefed up GPU or an "AI" chip.
A matrix of that size in single precision is 32TB, a better question is how do you store it?