A correct answer may not necessarily complete a task
Microsoft Learn explains in the Copilot Studio multi-round assessment documentation that this type of assessment is used to observe context-maintaining, clarification, and multi-step tasks. The document indicates that it applies to its standard harness experience, and it cannot be inferred that all tenants, product entrances, and modes are the same. ChinaOPC only borrows evaluation ideas, and the following examples do not rely on purchasing a specific tool.
For small teams that provide AI services, delivery testing can start when users change their minds midway. After the task changes, if the system still retains the old time or object, even if every reply is correct, it may continue to do the canceled work. Conversations, to-do status, and actual actions should be observed at the same time, and you cannot just rate the reply text.
Create a sample using a content scheduling conversation
All the following are simulated scenarios, there are no real customers, accounts or content to be released. In the first round, the user asked the system to schedule the three reviewed manuscripts for Friday; in the second round, one was replaced with a new manuscript; in the third round, it was changed to Saturday; in the fourth round, the system was told not to publish it yet but to keep the draft. The test is expected to form the latest draft schedule. The old draft and the old date are no longer valid and have not been released to the public.
Add another sentence, "Come on the one just now" and observe whether the system can recognize the reference. If there are two reasonable versions, the part to be confirmed should be clearly pointed out; if the context is clear enough, you can continue to sort it out without re-confirming every sentence. What needs to be checked here is whether there is a basis for understanding the task, rather than mechanical statistics of the number of questioning.
Each round leaves the currently valid requirements
The current round of input, still valid objects and times, canceled arrangements, current deliverables and allowed actions can be saved in the test record. When the system expresses "cancelled", it is necessary to check whether the task status and the subsequent queue are really stopped; if only the chat text is changed without stopping the pending actions, it cannot be recorded as passed.
- Replacement: whether new materials enter the current version and whether old materials are still misused.
- Rescheduling: whether all subsequent outputs will use the new time, and how to deal with the old schedule.
- Cancel: whether to stop actions that have not yet taken place and whether to keep the agreement draft.
- Recovery: Whether to indicate which version, scope and execution conditions are used when restarting.
For external actions that have taken place before the user cancels, the facts should be retained and the follow-up processing entry should be explained. Deleting the conversation cannot be equated with undoing the published content. If the first request is only for scheduling, it cannot be expanded by the system into actual release. Preparation, confirmation and execution correspond to existing authorization and business rules respectively.
Acceptance depends on the final state and what happened in the middle.
In addition to checking the last reply, you can also compare execution records and final deliverables. If the old version was sent out incorrectly and a correct draft was later generated, this use case should still record the problem in progress. For repeated messages, timeout retries, and user switching topics, check whether the system has repeated executions, lost cancellation status, or mistakenly accepted another task.
Prioritize the use of isolated environments and simulation materials for testing, and clarify which interfaces only return demonstration results. Save the model or configuration version, sample and failure reason, rerun the relevant dialogue after modification, and then compare whether the original problem is solved and whether it affects other tasks. Systems without actual execution capabilities can first verify the draft status without claiming to have completed external action testing.
When delivering, the measured range, untested actions, known limits and manual takeover entrance are provided to the customer. Assessment records can help both parties determine the next step, but they cannot guarantee that real conversations will never go wrong; after new business actions or permission changes, new tasks still need to be checked.
Extended reading: Sample boundaries of single-round Q&A testing; Review before AI content release; FDE delivery acceptance.