Even based on the technical marketing claims the consultants were coming in with, it was a product less capable than the IBM Prolog-based expert systems that I built out 15-20 years ago to triage and correlate network events in a large WAN infrastructure.... and that was a product with a whole slew of major implementation and operational problems!
That sounds like IBM Tivoli Enterprise Console (TEC). They wrapped a big Prolog-based DSL that the users wrote their correlation rules in, around the underlying Prolog engine. You could dip into the raw Prolog if you wanted, but it was considered an advanced user technique. The vast majority of "advanced" users used the set of cookbook Prolog functions the DSL was represented as, and even more users only ever touched a GUI that represented a tiny subset of that cookbook. People figured out what parts of Prolog the Prolog engine actually supported mostly through trial and error at first, before IBM finally published more details about it.
This rules-based approach ran into the same challenges of expert systems: beyond a certain threshhold number of rules, reasoning about the rules logic became dramatically more difficult. It didn't help that IBM never supplied support tooling for the underlying Prolog engine to aid in performance profiling, debugging, build support, etc.
The lesson I drew from that experience is if you supply your users with an embedded language in your app, make sure the advanced users get access to the kind of tooling full developers want.
It is interesting to see that the Watson-Jeopardy system uses Prolog [1]. I have to wonder how much of that was re-purposed to the chatbot project you observed. If it was, then its disappointment might be predictable; the range of possible problem spaces in a support setting is going to be larger than the relatively more constrained NLP of the Jeopardy format.
I've yet to see a good NLP system oriented towards technical software support, backed by a support knowledge base, that functions substantially better than current text searching and result ranking technologies, so I'd be very interested to hear about other people's positive results with applying machine learning to this area.
[1] https://www.cs.nmsu.edu/ALP/2011/03/natural-language-process...
We started with a really promising system that would let us call out to compiled python code and pull in some more capability. But by that time, we finished the original project and moved on.
Typically people are deploying chatbots as a cost cutting measure to reduce labor. Sometimes that means using the chatbot as a fancy IVR, getting customers to abandon the transaction or any of several other paths. Actually answering the questions doesn't always matter!