Comparing Models: Fix the Examples and Business Criteria First

First determine the tasks to be completed by the model

The official Langfuse evaluation document introduces the organization of the data set and evaluation. For entrepreneurs, it is first necessary to determine the results that customers need: extracting facts, complying with formats, citing information, or handling some kind of exception. Without task standards, it is easier to be left with language style preferences.

Fixed input, retain original output

Select samples that are approved for use and representative of the business, including insufficient data and difficult situations. Models, configurations, prompts and dates are recorded when comparing and judged against the same criteria by appropriate personnel. You cannot modify the input of one party while looking at the results, and then regard the performance under different conditions as a fair comparison.

Errors can be classified into factual inconsistencies, missing conditions, format failures, and inexecutables, and are handled in different ways. Also record delays, retries, and manual correction times; a good first-time answer does not mean a lower overall delivery cost.

The output should be the selection basis

Deliver samples, scoring instructions, failure lists and applicable boundaries, and do not package small-scale test tests into industry rankings. Revisit key examples after changes to the model or business data. Further reading: Model replacement risk.