Provider workflows
OpenAI Evaluation and Error Triage
Apply eval-driven development to an OpenAI workflow, preserve request evidence, classify failures, and release or recover through explicit gates.
- Format
- Playbook
- Level
- Intermediate
- Audience
- Developer, Operator, Leader
- Owner
- Project42 Editorial
- Review cadence
- Every 30 days
- Prerequisites
- A defined OpenAI workflow; Representative redacted or synthetic cases; Access to application telemetry without secrets
Use eval-driven development
Translate the real task into measurable success and failure criteria before changing prompts or models. Build representative, boundary, adversarial, refusal, and regression cases from redacted or synthetic data. Prefer deterministic grading for schemas, calculations, citations, required evidence, and prohibited outputs; calibrate human or model graders against examples when judgment is necessary.
Version the prompt, model, parameters, dataset, grader, and thresholds. Compare candidates with the current baseline across quality, safety, latency, and cost. Inspect slices and severe individual failures rather than relying on one average. Promotion requires all blocking thresholds and documented approval.
Classify the failure before acting
Separate connection, authentication, permission, invalid request, rate limit, server, model-completion, application-validation, and downstream-side-effect failures. Capture HTTP status, error code and type, x-request-id, response ID when present, timestamp, model, attempt, latency, and a privacy-safe input fingerprint. Never log the API key or sensitive body.
Do not retry authentication, permission, or invalid-request errors unchanged. Retry only transient rate, timeout, connection, and server failures within a bounded exponential-backoff budget with jitter. Respect provider headers and preserve the application operation identifier so retries cannot duplicate side effects.
Workflow/version: [NAME / COMMIT]
Model/configuration: [MODEL / PARAMETERS]
Dataset/grader: [VERSION / METHOD]
Required thresholds: [QUALITY / SAFETY / LATENCY / COST]
Observed results and worst slice: [METRICS]
Failure class: [CONNECTION | AUTH | INVALID | RATE | SERVER | COMPLETION | APPLICATION | DOWNSTREAM]
HTTP/error/request and response IDs: [REDACTED EVIDENCE]
Retry decision and budget: [DO NOT RETRY | ATTEMPT N OF LIMIT]
Rollback target: [LAST VERIFIED VERSION]
Release decision/approver: [PROMOTE | HOLD | REVERT / OWNER]Expected evidence and verification
Expected evidence is a versioned evaluation report and a reproducible, privacy-safe triage record. Verify dataset integrity, grader agreement, thresholds, regression slices, retry limits, redaction, request identifiers, and the final release decision. Re-run fixed regression cases after every incident.
If the gate fails or production regresses, halt promotion and restore the last verified prompt, model configuration, or application version. Reconcile uncertain downstream operations before retrying, then repeat the same evaluation and smoke checks. Escalate persistent provider failures with request IDs and a redacted timeline; close only when service and post-rollback evidence pass.