> Has anyone has tried searching for new basic operations, below the level of neural networks?
Perceptrons aren’t even a good analogue for biological neural networks. Each dendrite in and of themselves behave something like a multi layer perceptron. Back propagation doesn’t resemble human learning. Biological neurons also have a temporal activation function. Todays most successful ANNs seem focused on raw compute, but they may be missing some of the secret sauce that permits more interesting behaviors.