Outrageously Small Neural Networks: 6,616 tok/s on One Intel AMX Core [pdf]

I think we should revisit outrageously small neural nets.

I needed a cheap model that runs at over 10k token/sec on a single CPU core for some data processing. So I gave Anthropic claude code a pile of tokens to build one.

It made three discoveries that I thought were interesting:

1) One Intel AMX core can train a 3M active parameter MoE foundation model at 6,616 tok/s on 4.91B NVIDIA Nemotron tokens in a few days.

2) That model shows emergent in-context copying, positional analogies, and basic arithmetic after about 250M tokens.

3) The foundation model gives large gains in downstream SFT, and the training & eval loss keep going down all the way through 4.91B (and likely beyond).

Claude is not as good as a great MLE at debugging MoE. It made a bunch of bone headed mistakes, but it got there in the end.

I asked it to write a paper about it's work, and it produced this.

Claude Co-Authored Paper: https://huggingface.co/gdiamos/amx-reasoning-v1-instruct/blo...

I read through it and it sounds a bit LLMy, but the main points and experiment results are correct.

Some of the models are published on HF: https://huggingface.co/gdiamos/amx-reasoning-v1-instruct