Skip to contentSkip to content
AI agent

Re-test the assistant after you change your knowledge

Keep a list of real customer questions, see which ones depend on a source you just edited or removed, re-test only those, and find out which answers got worse before customers do.

Last updated

Changing one article can fix one answer and break another. A list of test questions tells you straight away. This page is for people who edit the knowledge base and want to know nothing got worse.

Before you start

  • You need the AI assistant and a knowledge base. See the AI assistant.
  • You need test questions. Add them under Knowledge → Quality with the words a good answer must contain, for example "How long do refunds take?" must mention "14 days". Up to 100 questions.
  • Questions are tested for real the first time you run them, and each one remembers how it did.

How do I see which questions a change could affect?

  1. Edit or refresh a source, or remove one.
  2. Open Knowledge → Quality.
  3. A yellow bar says how many test questions use a source that changed since they were tested. Each affected question shows "Source changed since the last test: <name>" or "Source removed since the last test: <name>".

A question counts as affected when the source it last answered from changed, or the source you named as the one it should cite changed. The sources are compared by the time of their latest saved version, so a refresh that changed nothing does not flag anything.

How do I re-test only those?

  1. In the yellow bar, choose Re-test these. Questions that were never tested are tested too.
  2. Read the result. A red box lists questions that got worse since the last test, meaning they passed before and fail now. A green line says how many failed before and pass now.
  3. Fix the source or the question's expected words, and run it again.

Run all at the top tests every question, as before. The history of scores shows every run, whichever way it was started.

What does a worse result tell me?

The answer no longer contains the words you expect, or no longer comes from the source you named, or the assistant now hands the question to your team. Open the question to see the answer it gave last time and the sources it used. Use Why this answer on real chats to see what customers were told.

Common problems

  • Nothing is flagged but I changed an article. The article may not have been used by any test question last time. Add a test question for it.
  • A question is flagged again and again. It cites a source that is edited often. Each re-test clears the flag until the source changes next.
  • "No test question uses a source that changed since it was tested." Everything is up to date. Choose Run all for a full check.
  • The test is slow or stopped. Each question is one AI call and counts against your AI budget.

Related

See replaying past answers for real questions from chats, the assistant safety check, and the AI assistant.

Frequently asked questions

Does a flagged question mean the answer is now wrong?

No. It means the question last used a source that has changed or been removed since. Re-testing finds out whether the answer still passes.

Does re-testing cost money?

Each tested question asks the assistant once, which is charged to your AI budget as a test. Re-testing only the flagged questions costs less than testing all of them.

How is a question judged?

By the words a good answer must contain, and the source it should come from if you named one. Questions that must be handed to a person pass when the assistant hands them over.