Architectural Effects on Maximum Dependency Lengths of Recurrent Neural Networks

https://arxiv.org/abs/2006.16236 Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

I've been working on using spiking neural networks (highly recurrent) to directly transform text via buffer mechanisms actuated by spiking activity. Ive had some marginal degree of success but there seems to be a bit of a wall. In these experiments, "thinking harder" takes the form of a growing pending spike queue. For practical reasons, this activity needs to be clipped or inhibited at some point. In real world biology, I suspect the power law growth in spiking activity is what contributes to much of cognition/intelligence. There's really no practical way to simulate it at full scale outside of hypothetical neuromorphic settings.

For now, I think we need to rely on probability, statistics and clever functions to drive most approaches forward. "Thinking harder" seems to require unconstrained resources if you need a decision in near real time.