Provider workflows
Cross-Provider Evaluation and Migration Workflow
Compare or migrate Anthropic, OpenAI, and Google workflows through one application-owned contract, provider-specific adapters, representative evaluation, staged cutover, and tested rollback.
- Format
- Playbook
- Level
- Advanced
- Audience
- Developer, Operator, Leader
- Owner
- Project42 Editorial
- Review cadence
- Every 30 days
- Prerequisites
- A versioned application success contract; Provider-specific adapters; A representative redacted or synthetic evaluation set
Compare outcomes without false equivalence
Freeze an application-owned input, output, evidence, safety, and latency contract, then map each provider through its own adapter. Record unsupported capabilities as gaps rather than forcing one API's tools, state, citations, refusals, or structured output into another provider's vocabulary. Use the same representative cases and scoring policy while preserving provider-specific diagnostics.
Measure task quality, severe failures, safety, evidence support, schema validity, tool correctness, latency distribution, cost, and operational failure rate. Version prompts and settings per provider; identical prompt text is not automatically a fair configuration. Calibrate judgment graders with blinded human examples and inspect slices, disagreements, and worst cases.
Stage migration and cutover
Run offline evaluation first, then shadow traffic with no user-visible or external side effects, then a bounded canary with stop conditions. Preserve provider, model, configuration, adapter, request IDs, scores, and redacted outcomes. Never send production data to a new provider before privacy, contract, region, retention, and authorization review.
Promotion requires every blocking threshold and named approval. Treat compatibility endpoints as partial adapters, not proof that semantics match. Maintain the previous adapter and configuration until the observation window and reconciliation checks complete.
Application contract/version: [NAME / VERSION]
Baseline provider/model/adapter: [VALUES]
Candidate provider/model/adapter: [VALUES]
Capability gaps and explicit exclusions: [LIST]
Dataset/segments/holdout: [VERSION]
Metrics and blocking thresholds: [QUALITY / SAFETY / EVIDENCE / TOOL / LATENCY / COST]
Offline result and worst slice: [EVIDENCE]
Shadow result: [EVIDENCE / NO SIDE EFFECTS]
Canary percentage/time/stop conditions: [PLAN]
Data-policy and security approval: [OWNER / DATE]
Cutover decision: [PROMOTE | HOLD]
Rollback trigger/target/owner: [CONDITIONS / LAST VERIFIED VERSION / OWNER]
Reconciliation and closure evidence: [CHECKS]Expected evidence and verification
Expected evidence is a reproducible comparison report and migration record showing provider-specific configurations, capability gaps, dataset version, grader agreement, thresholds, offline and staged results, approvals, and traceable cutover status. Verify no secret or sensitive data leakage, no hidden provider fallback, correct attribution, side-effect isolation, and complete rollback readiness.
Trigger rollback on any blocking safety, correctness, policy, latency, cost, or operational threshold. Route traffic to the last verified adapter and configuration, stop candidate side effects, reconcile uncertain operations, and rerun smoke and regression checks. Close only after rollback or cutover postconditions pass and the retained evaluation evidence can reproduce the decision.