That was the PS3 SPU. It was very fast for its time, but only for the small subset of code that could work within its constraints, and viciously difficult to program for.

We tried that, doesnt work out in general computers running more than one task.

DSPs do that, Atari Jaguar, Sony PS2 and PS3 did, all the GPUs manually manage cache.

More than one task and you start a fight over resources, have to manage hierarchies, priorities, all the stuff that now happens automagically.