AI Evaluation Plan Charter

Define the decision before the metrics

State the product decision the evaluation will support, the users and environments represented, the versioned system under test, and the credible harms of a wrong result. Separate task quality, groundedness, safety, privacy, tool behavior, latency, cost, accessibility, and operational reliability instead of hiding them in one average.

For each dimension, define an observable measure, baseline, target, blocking threshold, critical slices, sample-size rationale, grader, and acceptance authority. A model-based grader is another measured component: calibrate it against deterministic checks or blinded human judgments and record disagreement.

Record a reproducible evaluation charter

Use synthetic or authorized data, minimize retained content, and freeze test cases before comparing candidates. Keep a private holdout when repeated tuning could overfit the visible set. Assign owners for the dataset, graders, execution, risk review, decision, monitoring, and the next scheduled review.

Decision supported: [PROMOTE | HOLD | ROLLBACK | INVESTIGATE]
Use case and affected users: [BOUNDARY]
System under test: [APP, PROMPT, MODEL, TOOLS, DATA, CONFIG VERSIONS]
Baseline: [VERSION AND RESULT]
Dimensions: [QUALITY | GROUNDEDNESS | SAFETY | PRIVACY | TOOLS | LATENCY | COST]
Cases and critical slices: [REPRESENTATIVE, EDGE, ADVERSARIAL, ACCESSIBILITY]
Graders and calibration evidence: [CODE | HUMAN | MODEL -> AGREEMENT]
Blocking thresholds: [DIMENSION/SLICE -> MINIMUM OR MAXIMUM]
Stop criteria: [SEVERE FAILURE | DATA LEAK | INVALID RUN | BUDGET]
Owner and cadence: [DATASET, GRADER, RUN, RISK, DECISION -> ROLE/DATE]
Monitoring and rollback: [SIGNAL -> LAST VERIFIED VERSION]
Evidence location and retention: [AUTHORIZED STORE -> PERIOD]
Decision record: [RESULT, APPROVER, DATE, RESIDUAL RISK]

Expected evidence and verification

Expected evidence includes the frozen charter, versioned cases and graders, calibration results, complete run metadata, slice-level results, severe-failure review, and a signed promote, hold, rollback, or investigate decision. Verify that every stated objective maps to at least one case and that every blocking threshold is computed from captured results.

Stop when protected data is unauthorized, the evaluated system differs from the declared version, a blocking safety case fails, grader calibration falls below its declared minimum, or the run cannot be reproduced. Recovery means invalidating the affected run, preserving a redacted audit record, restoring the last verified system when exposure occurred, correcting the plan or fixture, and rerunning without changing thresholds after seeing results.