Ask HN: Train ChatGPT on a large set of functional analysis documents
The quality of documentation is exactly what you could expect from my first sentence.
We do not have much in terms of actual technical documents, because, hey, you can always open the source code in your editor and try to understand what the hell is going on.
What we do have, though, are a few thousands word doc documents that were used to by analysts to discuss features with the business users, and try to make it unambiguous enough to pass to the developers.
Quality is not that great here, either, but I just wonder if anyone had some experience with a similar situation.
What I hope to accomplish is being able to ask ChatGPT (or any other LLM system, I just mention ChatGPT as a shorthand here) questions like:
- What can you tell me about Feature X? There is a hardcoded limit of 3 X and I need to understand the implications if we remove it or make it a user-set parameter.
- What was purpose of Field XY on table WZ? It looks like it has been set to null for the last 5 years.
- Configuration parameter X is a bit obscure to me. What is it used for?
- Function X uses pessimistic locking. What should we change to make it use optimistic locking instead?
- We need to revamp feature Y due to new laws governing that part of the process. Specifically, we are now required to send an email describing the changes whenever FOO status becomes BAR. Can you suggest how to proceed with it, based on similar functionalities already implemented in the past?
As a sort of bonus question, I suppose we can also provide the source code for all the features described in the functional analysis (in the source code there are always comments defining which lines of code were added/removed/modified with the id of the analysis document requesting it) but unless it is absolutely vital I'd prefer to start by mining the human readable documents first.
[I am trying to create a business case for a pilot project on this space, so I would like to know if anyone has attempted this already, or if it is not even worth trying].