How do you know AI is telling your customers the truth?
I ask this question of every company that already has a chatbot. Almost nobody can answer.
In most such tools you see: 1,240 conversations, 78% "resolved", a 4.6 rating. Except that "resolved" most often means only that the customer didn't write a second time. It doesn't mean they got a correct answer.
A customer who got wrong information about delivery or warranty usually doesn't click the thumbs down. They simply disappear.
Tools show a list of questions the bot didn't answer. The problem starts later. Someone has to read that list, assess it, write a fix, upload it and check whether it works. During deployment someone still looks there. After six months, often nobody. And the bot keeps giving the same wrong answers.
Second thing: nobody measures the quality of answers. What is measured is the number of conversations, the percentage handed to a human, the customer rating. These are measures of whether the bot spoke. Not of whether it was right.
We do it differently.
First: before the agent answers its first customer, a set of control questions with correct answers is built from your documentation, price list and website. Someone from your company approves them. The agent takes this exam on the real system, and you see which questions passed, which didn't and why.
Second: underneath the agent that talks to customers works a second agent, trained for one task. Every day it reads all conversations and, instead of a list to review, leaves you a report: where knowledge was missing, where an answer was incomplete, what customers asked about that isn't in the base. To each finding it attaches a ready fix.
Third, and this is the difference: before the fix reaches you, it has already been tested on the control set. The test shows whether that question now gets a good answer and whether nothing else broke. You see the result and click "approve". The knowledge base updates itself, and the next day that question no longer comes back in the report.
I checked how it looks elsewhere. Most have a list of unanswered questions. Few propose ready fixes, usually in the most expensive plans. Testing a fix before deployment I found at nobody. Even the biggest write in their documentation that their agent doesn't test its proposals yet.
Five questions for every vendor, including us:
- How do you measure whether an answer was correct, not just whether there was one?
- Will I see the test result on my questions before launch?
- Who reads the questions the bot didn't answer, and when?
- What do I see after a change in the knowledge base: that it is better, or only that it is different?
- Is a fix tested before it reaches customers?
And if you hear "we have 95% accuracy", ask how it was calculated.
Want to see how the agent passes the exam on your website? Enter your website address and e-mail in the demo form. You get an agent that knows your site, and you ask it the questions your customers ask every day.