Changing one article can fix one answer and break another. A list of test questions tells you straight away. This page is for people who edit the knowledge base and want to know nothing got worse.
Before you start
- You need the AI assistant and a knowledge base. See the AI assistant.
- You need test questions. Add them under Knowledge → Quality with the words a good answer must contain, for example "How long do refunds take?" must mention "14 days". Up to 100 questions.
- Questions are tested for real the first time you run them, and each one remembers how it did.
How do I see which questions a change could affect?
- Edit or refresh a source, or remove one.
- Open Knowledge → Quality.
- A yellow bar says how many test questions use a source that changed since they were tested. Each affected question shows "Source changed since the last test: <name>" or "Source removed since the last test: <name>".
A question counts as affected when the source it last answered from changed, or the source you named as the one it should cite changed. The sources are compared by the time of their latest saved version, so a refresh that changed nothing does not flag anything.
How do I re-test only those?
- In the yellow bar, choose Re-test these. Questions that were never tested are tested too.
- Read the result. A red box lists questions that got worse since the last test, meaning they passed before and fail now. A green line says how many failed before and pass now.
- Fix the source or the question's expected words, and run it again.
Run all at the top tests every question, as before. The history of scores shows every run, whichever way it was started.
What does a worse result tell me?
The answer no longer contains the words you expect, or no longer comes from the source you named, or the assistant now hands the question to your team. Open the question to see the answer it gave last time and the sources it used. Use Why this answer on real chats to see what customers were told.
Common problems
- Nothing is flagged but I changed an article. The article may not have been used by any test question last time. Add a test question for it.
- A question is flagged again and again. It cites a source that is edited often. Each re-test clears the flag until the source changes next.
- "No test question uses a source that changed since it was tested." Everything is up to date. Choose Run all for a full check.
- The test is slow or stopped. Each question is one AI call and counts against your AI budget.
Related
See replaying past answers for real questions from chats, the assistant safety check, and the AI assistant.