For anyone getting up to date with language AI, taliesinb recommended this paper as "a real eye-opener" in the discussion about Google's Go AI
https://news.ycombinator.com/item?id=10984973 :
"A Cognitive Neural Architecture Able to Learn and Communicate through Natural Language"
http://journals.plos.org/plosone/article?id=10.1371%2Fjourna...
It uses a more complex (I daresay more advanced) model than the LSTM paper you linked, a model that can be trained to learn and use language constructs (in his words, "mimic certain kinds of child-level cognition with TINY amounts of training data"). Definitely worth a read.
This ANNABELL model doesn't require much training input to go from blank slate to conversation-capable (1587 input sentences -> 521 output sentences; see Appendix S1 for examples). This high learning efficiency might imply that it indeed resembles language-processing architecture in humans (which it was modelled after). At the low level it's also neurons, but organised and connected into specific, planned subsystems.
A key point is that the central executive (i.e. the core, which controls the flow of data between slave systems) is a trainable neural network itself, which learns to generate the "mental actions" that control the flow of data between slave systems (like short-term and long-term memory), rather than rely on fixed rules to control the flow. This apparently allows the system to generalise.