Skip to content
Guillermo AyalaAI consulting & development

Blog · AI for small and midsize businesses

Version control: stop an AI assistant citing an old policy

A new policy file does not automatically remove the old edition from an assistant. Version control needs an explicit lifecycle from approval through search-index update and verification. Test both current and historical questions so the assistant preserves the difference between present policy and archived evidence.

Two dated policy editions entering an index with only the current edition selected.
AI-generated editorial illustration.
In this article
  1. Define identity and status
  2. Use an update workflow
  3. Test a conflict deliberately
  4. Assign ongoing checks
  5. Related reading

A company replaces a policy in its shared folder, but the assistant continues quoting the old edition. The file changed; the searchable copy may not have. Preventing this requires a document lifecycle that covers approval, indexing, removal and verification.

Define identity and status

Give each policy a stable ID and each edition a version. Record owner, approval state, effective date and superseded version. A filename ending in final-new is not a reliable status rule. The business owner decides which edition governs; the technical team implements that decision in retrieval.

Minimum version-control fields
FieldPurpose
Policy ID and editionConnect related versions without confusing separate documents.
Status and effective dateDistinguish approved current, future and historical material.
Source ownerName who resolves conflicts and authorizes changes.
Index confirmationRecord when the change became searchable and was checked.

Use an update workflow

  1. The owner approves the edition and its effective date.
  2. The update process replaces or filters superseded searchable passages.
  3. Copies, cached results and derived summaries are handled consistently.
  4. A reviewer runs known questions and opens their cited sources.
  5. The change is confirmed, or the affected topic is paused pending correction.

Microsoft's RAG overview describes freshness and indexing concepts. Your implementation still needs a defined update mechanism; copying a file into a folder is not proof of successful retrieval.

Test a conflict deliberately

In a hypothetical company, a reimbursement procedure changes its submission deadline. Create synthetic editions with different deadlines and mark their dates clearly. Ask for the current process, the process before the change and a question that omits the relevant date.

The current answer should use the edition in force. Historical questions should use archived material only where that capability is authorized and designed. Ambiguity should prompt clarification. Do not leave conflicting editions available and expect the model to infer authority from persuasive wording.

Assign ongoing checks

Measure the interval between approval and verified availability, failed update checks and answers citing superseded material. Review duplicate repositories and unofficial copies. The policy owner resolves meaning; the technical owner investigates synchronization.

The Microsoft evaluation guide supports testing individual stages. Keep stable questions for each major policy so an update can be checked against prior behavior.

An update is complete only when the expected current answer and the disappearance of superseded evidence have both been checked. Include a negative test that searches for the old wording. If historical access is supported, keep it in a separately defined route with dates. Record when synchronization is delayed and tell users when the affected knowledge cannot yet be treated as current.

Include the relevant business date explicitly when current and historical answers legitimately differ.

Bring two conflicting editions and a few current-versus-historical questions to a knowledge assistant review to design an update process your team can maintain.

Back to the blog · Practical AI guides