> wonder why we don't have external "neural processing" devices like we once had soundcards.

Some video cards/GPUs have become just that, becoming more and more geared towards non-graphics workloads ...