* The use of tree-based knowledge extraction with manual review + the graph of the resulting information by principle component extraction demonstrates the effective base of the context.
* The use of a Sentence-BERT model specifically for tool matching avoids the hallucination problem of LLMS offering fake solutions/diagnosis steps.
* The tree-based multi-LLM-expert diagnosis by vote system also addresses hallucination and failures like looping through the same solutions over and over in complex cases, and is reminiscent of the monte-carlo advance for AlphaGo and paxos consensus protocols. AND it provides output in an auditable way, which is important for incidents.
When testing, they evaluate against a human DBA with two years of experience, which seems kind of junior to me. Notably, in the results the D-Bot usually (9/12 cases) comes close to the junior DBA, but does not exceed it. However, the D-Bot definitely exceeds the results of raw LLM prompting and it has the obvious speed advantage over a human.
Overall, this gives me confidence that some of the LLM projects at my own company can be useful, since auditability + specific knowledge extraction are relevant to our work.