How to Prepare a Knowledge Base for an AI Support Agent

Before an AI support agent goes live, the groundwork is mostly administrative: a small set of current, approved documents, a named owner for each, and a list of representative questions that includes the difficult ones. The point is not to feed the agent everything you have, but to decide in advance which questions it can answer, which actions it may take, and when a person takes over. That boundary is what makes answer quality checkable.

A practical way to organise the material is one record per source. Each record carries a topic, an effective date, the person who maintains it, and the answers or actions it permits. When two records disagree, or a question falls outside them, the agent needs an agreed fallback instead of an invented answer. Naming the owner matters because policies change, and the effective date is how you notice when a document has quietly stopped being true.

Consider a labelled example. Suppose your refund policy was revised last quarter, but the old version still sits in the shared drive alongside the new one. An agent given access to both versions may answer some customers with the expired terms. The fix is specific: remove or mark the outdated document, update the record, then retest the same refund question until the answer comes from the current source only. Platform mechanics make retesting worthwhile even when the setup looks right. Microsoft’s Copilot Studio documentation describes the ungrounded-response setting precisely: turned off, it blocks any response generated in a turn where the agent used neither a knowledge source nor a tool, so a follow-up the agent thinks it can answer from earlier context is blocked as well. It is not a semantic check on whether an answer rests only on general knowledge, and the documentation notes the model may still blend general knowledge into a grounded answer. A correct answer can occasionally be withheld when the model fails to attach its citation.

Once the agent is running, measure what actually happened rather than what the dashboard implies. Intercom defines a resolution in two forms: a confirmed one, where the customer replies affirmatively, and an assumed one, where the customer simply leaves after the agent’s last answer without asking for more. On their billing model, an assumed resolution is still a counted outcome, and it is deducted if the customer returns to the same conversation for further help. For a UAE team serving Arabic and English speakers, that means tracking reopened cases next to correct answers, and testing representative questions in both languages so the policy meaning matches. A customer who leaves satisfied and a customer who gives up look identical in the log, and a silent departure may never reopen at all. Assumed resolutions alone cannot establish satisfaction; weigh reopened cases alongside affirmative feedback and a review of answer quality before trusting the number.