Models and providers
Hosted vs. Self-Hosted Model Decision
Choose a managed API, cloud deployment, managed compute, or self-hosted model from workload, data, reliability, staffing, and total-cost evidence.
- Format
- Decision Path
- Level
- Advanced
- Audience
- Developer, Operator, Leader
- Owner
- Project42 Editorial
- Review cadence
- Every 60 days
- Prerequisites
- A workload and data-classification statement; Expected traffic, latency, availability, and recovery targets; An owner for model and infrastructure operations
Separate access model from infrastructure model
Managed provider APIs minimize model-serving operations but retain provider availability, quota, lifecycle, and data-boundary dependencies. Cloud catalog deployments add tenant, region, network, identity, and capacity controls. Managed compute and self-hosting add image, weights, hardware, scaling, patching, observability, safety, and incident ownership.
Local execution can support disconnected, low-latency, or tightly controlled workloads, but locality alone does not prove privacy, security, quality, or lower cost. Include model license, weight provenance, runtime dependencies, accelerators, idle capacity, engineering labor, power, monitoring, backups, upgrades, and recovery.
Model the real workload and failure boundary
Evaluate the same workload across viable deployment shapes. Include cold start, sustained and burst throughput, tail latency, input and output size, tool calls, safety controls, regional availability, data retention, egress, support, and the cost of keeping a safe fallback.
Choose self-hosting only when the team accepts the operational ownership and the measured benefit clears a declared threshold. Choose hosted service only when its data, identity, lifecycle, quota, and recovery constraints meet the requirement.
Task: [WORKLOAD AND HOSTING DECISION]
Scope: [DATA, USERS, REGIONS, MODELS, AND DEPLOYMENT OPTIONS]
Permissions: [DATA ACCESS, CLOUD/TENANT, MODEL LICENSE, AND DEPLOY AUTHORITY]
Demand: [NORMAL/BURST CONCURRENCY, TOKENS, LATENCY, AND AVAILABILITY]
Option costs: [USAGE, COMPUTE, STORAGE, EGRESS, LICENSE, LABOR, AND IDLE CAPACITY]
Control boundary: [IDENTITY, NETWORK, DATA, WEIGHTS, RUNTIME, AND LOGS]
Operations: [SCALING, PATCHING, MONITORING, INCIDENTS, AND SUPPORT]
Stop conditions: [LICENSE GAP, DATA VIOLATION, UNSUPPORTED SLO, OR UNOWNED OPERATIONS]
Verification: [REPRESENTATIVE LOAD, QUALITY/SAFETY GATE, FAILURE TEST, AND TCO]
Recovery: [VERIFIED FALLBACK ROUTE, STATE RECONCILIATION, AND RESTORE TEST]Expected evidence and verification
Expected evidence includes the data boundary, demand model, exact model and runtime artifacts, license, deployment options, representative quality and safety results, performance distribution, total-cost assumptions, operational owner, failure tests, and an expiring decision.
Verify a controlled failover or restore, not just successful inference. If the selected shape violates a data boundary, loses required capacity, fails critical evaluation, or exceeds the accepted operating burden, stop new traffic and route to the recorded fallback while state and side effects are reconciled.