Why disagreement should survive the first pass

Consensus is useful. But a clean majority can also hide the exact place where a model, rubric or product assumption is weakest.

Many evaluation workflows are designed to remove disagreement as quickly as possible. Collect several labels, resolve conflicts, produce the final answer. That is sometimes necessary. It is not always the most informative thing to do first.

Different answers can mean different things

Reviewers may diverge because the task is ambiguous. They may have different legitimate preferences. One reviewer may have missing context. A rubric may contain an unstated boundary. A model output may be acceptable under one interpretation and poor under another.

If the workflow immediately collapses those responses into one label, the underlying cause disappears.

The useful unit is the evidence around the answer

A stronger evaluation can retain the task version, independent responses, relevant rationale, QC state and later ground truth where available. The goal is not to celebrate disagreement. It is to know when disagreement carries information.

What this changes

The design question becomes: which differences should be reconciled, which should be segmented, and which should trigger another question? That is more useful than assuming the first majority is the end of the evaluation.