Models and providers
Role-Based Model Routing Evaluation
Assign models to planner, executor, reviewer, researcher, and specialist roles only after testing the role contract and end-to-end workflow.
- Format
- Playbook
- Level
- Advanced
- Audience
- Developer, Operator
- Owner
- Project42 Editorial
- Review cadence
- Every 45 days
- Prerequisites
- Explicit role and handoff contracts; Representative end-to-end workflow cases; Per-step and terminal quality, safety, latency, and cost measures
Define the role before choosing its model
A planner needs decomposition and constraint fidelity; an executor needs tool-contract accuracy and recovery behavior; a reviewer needs defect recall, evidence, and independence; a researcher needs source discipline; a specialist needs domain-specific acceptance criteria. Define required inputs, outputs, forbidden actions, budgets, and handoffs for each role.
Do not assign a provider or model from reputation alone. A less expensive model can be appropriate for a narrow structured transition while failing an ambiguous planning task, and a stronger model can add cost without improving a deterministic step.
Evaluate transitions and terminal outcomes
Test each candidate on isolated role cases, then run the complete multi-role trajectory. Score instruction adherence, structured output, tool arguments, uncertainty, handoff completeness, reviewer defect detection, terminal correctness, safety, latency, and total cost.
Compare against a simple one-model baseline. Routing complexity is justified only when it improves declared outcomes enough to pay for more failure modes, state transitions, observability, and recovery paths.
Task: [WORKFLOW AND ROUTING HYPOTHESIS]
Scope: [ROLES, CANDIDATE MODELS, TOOLS, DATA, AND ENVIRONMENT]
Permissions: [ROLE-SPECIFIC READ/WRITE/TOOL/AUTHORITY LIMITS]
Role contract: [INPUT -> REQUIRED OUTPUT -> FORBIDDEN ACTION -> HANDOFF]
Baseline and candidates: [ONE-MODEL BASELINE -> ROUTED OPTIONS]
Cases and metrics: [ROLE CASES + END-TO-END CASES -> THRESHOLDS]
Budgets: [TOKENS, TIME, COST, RETRIES, AND TOOL CALLS]
Stop conditions: [CRITICAL FAILURE, INVALID HANDOFF, LOOP, OR BUDGET]
Verification: [REPEATED RUNS, TRANSITION REVIEW, TERMINAL POSTCONDITION]
Recovery: [FALL BACK TO VERIFIED ROUTE, CHECKPOINT STATE, AND REPLAY SAFELY]Expected evidence and verification
Expected evidence includes versioned role contracts, exact model configurations, a one-model baseline, isolated and end-to-end results, handoff and tool trajectories, slice metrics, budgets, and a routing decision with owner and expiry. Verify the terminal user outcome as well as each role's score.
Stop when a role exceeds authority, a handoff loses required context, the loop cannot converge, a critical case regresses, or routing cost exceeds its threshold. Recover by restoring the last verified route and prompt set, reconciling external actions, and replaying only idempotent or compensated transitions.