Dammit, why do I always get carried away discussing details and forget the main point?

That, what you say- that's the business. An RNN (any ANN) is not a human brain. It is one model of the human brain, based on our very limited understanding of human brains. It doesn't matter what ANNs can and cannot do, how well or bad they perform, it tells us nothing much about how we do language; or vision, motion, decision-making, anything.