Context and Quality Regression Triage

Freeze the observation before changing the system

Capture a privacy-safe reproducer and the exact application, prompt, policy, context recipe, retrieval corpus and index, model, parameters, tools, grader, dependency, and deployment versions. Compare the same case against the last verified baseline. Do not change prompt, model, context, retrieval, and grader together; one-at-a-time experiments preserve causal evidence.

Classify the first divergence as task-distribution shift, missing or stale context, retrieval miss, conflicting evidence, context truncation, instruction collision, prompt regression, model behavior change, tool trajectory change, parser or post-processing defect, grader drift, or telemetry defect. Inspect critical slices and individual severe failures because an average can remain stable while one population or use case regresses.

Create a regression comparison record

Keep source and candidate runs under equivalent conditions and record unavoidable differences. Turn a confirmed failure into a governed regression case with provenance, expected and forbidden outcomes, and retention controls. If the affected content is time-sensitive, reverify authoritative sources rather than teaching the model to repeat stale text.

Observed regression and impact: [SYMPTOM -> USERS/WORKFLOWS]
First-seen window: [UTC RANGE]
Baseline/candidate versions: [APP/PROMPT/CONTEXT/RETRIEVAL/MODEL/TOOLS/GRADER]
Reproducer and slice: [AUTHORIZED CASE -> EXPECTED/FORBIDDEN]
First divergent transition: [CONTEXT | RETRIEVAL | MODEL | TOOL | APP | GRADER | TELEMETRY]
Controlled comparison: [ONE VARIABLE -> RESULT]
Evidence freshness: [SOURCE/CORPUS/INDEX -> VERIFIED DATE]
Owner and cadence: [COMPONENT OWNER -> UPDATE INTERVAL]
Stop criteria: [SAFETY | PRIVACY | CRITICAL SLICE | INVALID COMPARISON | UNKNOWN CAUSE]
Recovery or rollback: [RESTORE VERIFIED VERSION | DISABLE FEATURE | DEGRADE SAFELY]
Verification: [FULL SET + CRITICAL SLICES + PRODUCTION-SHAPED CHECK]

Expected evidence and verification

Expected evidence includes the minimized case, exact versions, first divergent transition, controlled comparisons, slice-level metrics, source freshness, root-cause confidence, containment, owner, and a durable regression test. Verify the proposed fix against the failing case, full representative set, critical safety and accessibility slices, grader calibration, latency, cost, and a production-shaped smoke test.

Stop rollout when a blocking safety or privacy case fails, a critical slice crosses its threshold, baseline and candidate conditions are not comparable, evidence is stale, telemetry is unreliable, or the cause remains unknown at consequential scale. Recovery means restoring the last verified component set or disabling the affected route, invalidating misleading results, rebuilding context or indexes when required, and promoting only after the unchanged gate passes.