In the text generation tool outlined above (and indeed many of the convnet-based visual networks), the hidden layers are there precisely to extract 'meaning'. The lower layers (closer to the source input) deal with syntax and feed upwards to hidden layers that extract semantic features, which in turn feed upwards to more layers, each with a bigger overview of the semantic features and thus ultimately the context. That's the idea anyway.