Evaluation and safety
Human-Controlled AI Release Gate
Convert evaluation, safety, security, privacy, operational, and rollback evidence into an accountable promote, hold, or reject decision.
- Format
- Checklist
- Level
- Advanced
- Audience
- Developer, Operator, Leader
- Owner
- Project42 Editorial
- Review cadence
- Every 60 days
- Prerequisites
- A versioned release candidate and evaluation report; Named product, risk, and operational owners; A tested rollback or feature-disable path
Require evidence from every affected boundary
Bind the decision to exact application, prompt, model, retrieval, tool, policy, data, infrastructure, and grader versions. Require representative and critical-slice evaluation, severe-failure review, security and privacy checks, accessibility evidence, capacity and cost limits, observability, incident ownership, dependency status, migration impact, and a rehearsed rollback.
Keep the approval human and role-based. Automated checks may block a release or assemble evidence, but they must not invent exceptions or approve a consequential change. Record waivers with scope, expiration, compensating controls, and accountable acceptance; never convert a failed blocking criterion into an undocumented average.
Run the promote, hold, or reject gate
Use staged exposure when consequence or uncertainty warrants it. Define canary population, duration, health signals, safety signals, error budget, cost ceiling, manual kill path, automatic stop conditions, and the evidence required for each expansion. Preserve the previous verified configuration until post-release closeout.
Release candidate: [APP, PROMPT, MODEL, TOOL, DATA, POLICY VERSIONS]
Evaluation: [REPORT -> ALL BLOCKING THRESHOLDS PASS]
Critical slices and severe failures: [RESULT -> REVIEWER]
Security, privacy, safety, accessibility: [EVIDENCE -> OWNER]
Operations: [SLO, ALERTS, RUNBOOK, CAPACITY, COST, SUPPORT]
Data and migration: [COMPATIBILITY, RETENTION, REVERSIBILITY]
Canary plan: [POPULATION, DURATION, EXPANSION EVIDENCE]
Owner and cadence: [PRODUCT, RISK, OPERATIONS -> GATE/REVIEW DATE]
Stop criteria: [SAFETY FAILURE | ERROR BUDGET | COST | UNKNOWN SIDE EFFECT]
Rollback and recovery rehearsal: [METHOD -> VERIFIED RESULT]
Waivers: [SCOPE, OWNER, EXPIRATION, CONTROL | NONE]
Decision: [PROMOTE | HOLD | REJECT]
Approvers and timestamp: [PRODUCT, RISK, OPERATIONS]
Closeout evidence: [POST-RELEASE WINDOW -> RESULT]Expected evidence and verification
Expected evidence is an immutable decision record tied to exact artifacts, green blocking thresholds, documented residual risks, named approvers, successful rollback rehearsal, canary observations, and post-release closeout. Verify the deployed hashes and configuration match the approved candidate, alerts reach an on-call owner, the disable or rollback control works, and audit evidence excludes secrets and unnecessary user content.
Hold or reject when any blocking threshold fails, required evidence is missing or stale, an owner is absent, a dependency is degraded, rollback is untested, the canary exceeds a stop condition, or actual configuration differs from approval. Recovery means halting expansion, disabling the affected capability or restoring the last verified version, reconciling data and side effects, communicating impact, rerunning the same gate after remediation, and retaining both decisions.