Is it naive to assume that we could have bots actually read the code and come up with an "understanding" of what the code does instead of the metadata written by a human for humans on the side ?

Yup.

Were using LLM retrieval methods to build Q&A bots at work. These are all fed with documents (user guides, release notes, transcribed videos etc).

Its still very much POC but the interesting thing is people seems to care about documents again a bit more knowing that it will be used in this manner.

I was thinking about developing something that rewards document producers if their response is cited and used successfully - would help strengthen the feedback loop.

Agreed, Chatbots will make it a lot easier for the discovery process of documentation.