The decision
Reliability starts with the documents the system is allowed to use and the questions it should decline. Retrieval can bring relevant company material into an answer, but finding a passage does not establish that it is current, applicable or sufficient. Give every source an owner and a revision process.
A worked example
For an illustrative supplier-policy assistant, ask about a return where the item is damaged and the purchase date is unclear. The answer should cite the applicable policy, identify the missing date and route an unresolved exception to a person. It should not invent a deadline to make the conversation feel complete.
Alternatives worth weighing
A searchable knowledge base may be enough when users want exact documents. A retrieval-based assistant helps connect language to passages. A structured lookup is preferable for an authoritative order status. Combining them is useful only when the interface makes the source of each answer clear.
Where the plan breaks
Build an evaluation set containing ordinary questions, outdated documents, conflicting policies, unavailable evidence and requests beyond the user's permissions. Check evidence selection separately from answer wording. A fluent answer with a decorative citation can still be unsupported.
Give a policy change an operational path
Consider what happens when the returns policy changes while the assistant is already in use. The owner supplies the approved document, effective date, audience and replacement relationship. The ingestion process should preserve a stable document reference and a revision identifier, then check that headings, exceptions and tables remain understandable after extraction. Updating the file alone is insufficient if old passages remain searchable. Plan how the previous version leaves the active index and how cached answers are invalidated.
For the fictional supplier assistant, separate public terms from internal exception guidance. Access should be enforced before passages enter the answer context, rather than asking the model not to repeat private material it has already received. A user changing roles should lose access through the same application permission rules. The design discussion should assign ownership of the source, retrieval configuration and escalations separately; those responsibilities may belong to different people.
Accept the assistant by its failure behavior
Build a small, named collection of questions with the policy owner. Include an ordinary return, an exception requiring approval, an ambiguous purchase date, an obsolete policy and a request for another customer’s information. Write the acceptable evidence and behavior for each case before changing the prompts. “Escalates because the date is missing” is a useful expected result; a confident invented deadline is not a near miss.
During review, examine the selected passage and answer together. A correct-looking response supported by the wrong policy should fail the check. If a new index revision regresses a known case, operators need a way to return to the previous tested configuration or temporarily offer document search. Keep enough version information to investigate complaints, with restricted access and a deliberate retention policy. Do not copy every private conversation into unrestricted analytics merely to make evaluation convenient.
Before you commission the work
Who updates the source when a policy changes? What should the system say when evidence is missing? Who owns escalations? Resolve those operational questions with an AI integration partner before judging the chatbot by a successful demonstration.