Blog · AI for small and midsize businesses
How to tell whether a chatbot helps: resolution, quality and handoff
A conversation ending inside the chatbot does not prove the customer got help. Measure resolution, answer accuracy and the experience of reaching a person together. Otherwise a dashboard can reward an assistant that gives confident answers while leaving more work for customers.

In this article
A chatbot scorecard should show whether customers achieve their service goals. Containment only describes conversations that did not reach a person. It can include successful answers, abandoned sessions and customers who gave up. Treating all three as resolution creates the wrong incentive.
Define the outcome before counting it
The service owner should specify what counts as resolution for each supported intent. A policy question may be resolved by an accurate answer the customer finds useful. An order-status query requires a successful authorized lookup. A refund request may be properly handled by a complete transfer rather than a final decision inside the bot.
Record unknown outcomes explicitly. A silent customer is not a confirmed success. Link later contacts to the same issue where your data and permissions allow it, and distinguish a new request from a customer reopening an unanswered one.
A balanced review sheet
- Task result: completed, transferred appropriately, unresolved or unknown.
- Answer quality: correct evidence, preserved conditions and no unsupported promise.
- Customer effort: repeated questions, unnecessary steps and ease of requesting a person.
- Transfer quality: correct destination, usable context and accepted ownership.
- Operating effort: staff review, corrections and maintenance needed to deliver the result.
Review both the retrieved evidence and the final explanation. Microsoft’s RAG guidance recommends evaluating process steps and the end result separately. Good retrieval can still produce a misleading answer; an apparently helpful answer may rely on the wrong document.
Check what the resolution rate includes
Consider a hypothetical week with 100 supported requests. Seventy never reach a person. After review, 45 appear resolved, 15 are unresolved and 10 have unknown outcomes. Reporting 70% resolution would erase those differences. The remaining 30 may include excellent transfers; reaching a person does not necessarily mean the chatbot failed.
Inspect a selection from every outcome, including apparent successes. If you only audit complaints, you cannot see incorrect answers customers accepted. Use reviewers who know the policy and require a reason for each failed answer, with its supporting case reference.
Read results by intent and language
Separate basic information requests from account-specific or exception-heavy cases. Also inspect Spanish and English, business hours and after-hours service. Changes in the mix can move the overall percentage even when no individual workflow improves.
Agree on review and stop conditions before a wider rollout. An unauthorized disclosure or recurring incorrect commitment deserves a different response from an awkward sentence. Assign each recurring failure to the owner who can change the policy, integration or conversation design.
Compare a stable period before and after an improvement, while documenting changes in volume and scope. Keep the scorecard small enough that someone reads the underlying cases. Use it to guide service decisions.
We can define a practical evaluation scorecard for the questions your assistant is actually expected to handle.