Evidence-Based AI Evaluation Rubric

Score observable evidence

Split the desired outcome into non-overlapping criteria such as correctness, evidence support, instruction fulfillment, safe tool behavior, privacy, clarity, and recovery. Define each level with inspectable evidence rather than adjectives. Mark non-compensable failures—such as a secret disclosure or unauthorized side effect—as blocking even when the weighted total would pass.

Choose the least subjective reliable grader. Use deterministic validation for schemas and postconditions, domain-qualified humans for consequential judgment, and model graders only after agreement is measured against a reviewed reference set. Hide candidate identity when possible and randomize comparison order to reduce bias.

Create rubric and calibration records

Maintain at least one complete passing anchor, one clear failing anchor, and boundary examples where graders commonly disagree. Version criteria, weights, prompts, grader models, and anchors together; changing a rubric after seeing candidate results creates a new evaluation.

Rubric version and use case: [ID -> SCOPE]
Criterion: [ONE OBSERVABLE DIMENSION]
Evidence inspected: [OUTPUT FIELD, SOURCE, TRACE, OR POSTCONDITION]
Score 0 anchor: [OBSERVABLE FAILURE]
Score 1 anchor: [MAJOR GAPS]
Score 2 anchor: [MINIMUM ACCEPTABLE]
Score 3 anchor: [STRONG RESULT]
Weight or aggregation: [RULE]
Blocking condition: [FAILURE THAT CANNOT BE AVERAGED AWAY]
Grader: [CODE | QUALIFIED HUMAN | MODEL/VERSION]
Calibration set and agreement target: [CASES -> METRIC/THRESHOLD]
Owner and cadence: [ROLE -> REVIEW DATE]
Stop criteria: [LOW AGREEMENT | MISSING EVIDENCE | CONFLICT OF INTEREST]
Recovery: [PAUSE DECISION -> RE-CALIBRATE -> RESCORE ALL AFFECTED RUNS]

Expected evidence and verification

Expected evidence includes the versioned rubric, weights and blocking rules, complete anchors, calibration cases, independent grader results, agreement metrics, disagreement adjudication, and traceable scores for each evaluated case. Verify arithmetic, test rubric instructions on unseen examples, and confirm that the same evidence yields acceptably consistent outcomes across graders and ordering.

Stop scoring when evidence is missing, instructions are ambiguous, agreement falls below the declared threshold, the grader can identify a candidate when blinding is required, or a rubric change would affect the decision. Recovery means freezing the decision, revising and independently reviewing the rubric, recalibrating, rescoring every affected candidate on the same cases, and retaining the superseded result for audit.