Research focuses on unwritten business rules
The FDE-Bench preprint introduces 49 tasks from customer projects, which are evaluated on requirements clarification and iterative delivery. The experiment has the agent obtain missing information from a simulated customer, and the results are checked by a task evaluator. This article only discusses the working methods based on this, and does not interpret the experimental results as a ranking of the capabilities of all models or all enterprise projects.
This setting reminds us that the customer’s initial words often contain the industry’s default premise. Even if the agent strictly follows the written instructions, it may still make arrangements that are unacceptable to the business. The information that needs to be supplemented is not the more the better, but the rules that will change the solution selection and acceptance results.
First find a specific example that will cause disagreement
The following is a simulation scenario designed by ChinaOPC: the customer wants to merge contacts from the same company, but the sales staff actually needs to distinguish between the headquarters and different branches. If you only remove duplicates based on the company abbreviation, the result may look neat, but the contact relationships between different regions may be lost. This is not an original assignment in the paper, but a worked example used to illustrate the problem.
Instead of continuing to ask "what else is required?" the client can be asked to judge whether two specific records should be merged and what exceptions are allowed to remain. Each time you get an answer, record the conditions under which it applies, who has the authority to confirm, and what actions it changes. When encountering conflicting statements, first find the corresponding position and scenario.
Convert the Q&A results into a set of materials that can be checked repeatedly
Confirmed rules can be made into short samples: original input, expected processing, exception reason and final confirmer. Then prepare records that are superficially similar but have different results, and check whether the system only remembers keywords. In this way, what remains of the requirements discussion are executable verification conditions, not just meeting minutes.
When the rules change, save the old conditions and the reason for the change, and rerun the affected examples. New information may indeed emerge from the field, and the delivery team should be able to explain "why the results are different this time than last time" rather than explaining all differences with model randomness.
After the clarification is completed, it is still necessary to verify whether the solution can be used
Knowing the rules does not guarantee that the code will be implemented correctly, the data will be complete, or the runtime will be acceptable. A small team can break the acceptance into three questions: whether the rules are confirmed by the business party, whether the results comply with the rules, and whether users can use and take over according to the established steps. Each question requires corresponding materials and cannot be replaced by a screenshot of a successful run.
The paper uses specific tasks, simulated customers and experimental settings, which cannot be directly converted into local customer satisfaction. For actual projects, it is more valuable to retain the correspondence from questions, answers to results. You can combine FDE delivery acceptance and Project delivery record to organize your own pilot materials.