Provider workflows
Anthropic Evaluation and Error Triage
Turn a Claude workflow into measurable tests, preserve request evidence, classify API failures, and release or recover through explicit gates.
- Format
- Playbook
- Level
- Intermediate
- Audience
- Developer, Operator, Leader
- Owner
- Project42 Editorial
- Review cadence
- Every 30 days
- Prerequisites
- A defined Claude workflow; Representative redacted or synthetic cases; Access to application telemetry without secrets
Define success before tuning
Write specific success criteria for the real task, then assemble representative, boundary, adversarial, refusal, and regression cases. Prefer deterministic checks for schemas, citations, calculations, prohibited content, and required evidence; use calibrated human or model graders only where judgment is necessary. Keep a holdout set and record the prompt, model, configuration, dataset version, grader, and threshold.
Compare changes against the current baseline on quality, safety, latency, and cost. Averages can hide severe failures, so inspect slices and worst cases. Promotion requires the agreed threshold with no blocking regression; changing the grader or dataset creates a new baseline.
Classify the failure before acting
Separate transport, authentication, permission, validation, rate-limit, overload, model-completion, application-validation, and downstream-side-effect failures. Capture the HTTP status, error type, request_id or request-id header, timestamp, model, attempt, latency, and a privacy-safe input fingerprint. Do not log keys or sensitive request bodies.
Treat 401 and 403 as credential or authorization problems, 400 and 413 as request problems, 429 as rate or acceleration limits, and 5xx as transient provider or infrastructure failures subject to the provider's current guidance. Retry only transient classes with bounded backoff and jitter.
Workflow/version: [NAME / COMMIT]
Model/configuration: [MODEL / PARAMETERS]
Dataset/grader: [VERSION / METHOD]
Required thresholds: [QUALITY / SAFETY / LATENCY / COST]
Observed results and worst slice: [METRICS]
Failure class: [TRANSPORT | AUTH | VALIDATION | RATE | OVERLOAD | COMPLETION | APPLICATION | DOWNSTREAM]
HTTP/error/request ID: [REDACTED EVIDENCE]
Retry decision and budget: [DO NOT RETRY | ATTEMPT N OF LIMIT]
Rollback target: [LAST VERIFIED VERSION]
Release decision/approver: [PROMOTE | HOLD | REVERT / OWNER]Expected evidence and verification
Expected evidence is a versioned evaluation report plus a triage record that another operator can reproduce without private data. Verify dataset integrity, grader calibration, required thresholds, regression slices, retry limits, redaction, request identifiers, and the final release decision.
If a release fails its gate or production signals regress, stop promotion, restore the last verified prompt, model configuration, or application version, and rerun the same evaluation. Reconcile any uncertain downstream operations before retrying. Escalate persistent provider failures with the request ID and redacted timeline; close only after post-rollback checks pass.