MODEL INFRASTRUCTURE
Choose the model, control the provider, and plan for failures. Explore the infrastructure between your application and its model APIs.
6 editorial selections · Sources reviewed · Our methodologyThese products have different scopes. This is a sourced starting point, not a performance ranking.
6 of 6 routers and gateways
Hosted gatewayAccess models through one API and control which inference providers serve a request. Provider preferences and fallbacks help manage availability.
Before you integrateCheck provider-level data policies and model parameter support when setting routing preferences.
Self-hosted gatewayRun a proxy or use a Python router to distribute requests across model deployments, with load balancing, retries, cooldowns and fallbacks.
Before you integratePlan deployment, provider credentials and shared usage tracking when running the proxy across multiple instances.
AI gatewayUse a unified API with conditional routing, provider and model fallbacks, caching and guardrails.
Before you integrateReview the configuration and deployment options that apply to your team's governance requirements.
Hosted gatewayRoute requests across providers and fallback models with visibility into routing attempts, latency, token usage and cost.
Before you integrateSet provider preferences explicitly and inspect response metadata to confirm which provider served the request.
Hosted gatewayManage model requests with logging, caching and rate limits, plus fallback providers for errors or configured timeouts.
Before you integrateDefine error and timeout behavior, and verify the authentication requirements of the endpoint you use.
Serve and evaluate learned routers that choose between a stronger and a weaker model using a configurable cost-quality threshold.
Before you integrateCalibrate thresholds on representative queries and evaluate the model pair you actually intend to deploy.
Prompt, context and requirements
Model policy, provider preference or fallback
The selected provider runs the model
An LLM router selects the model or inference provider that receives a request. Routing may follow explicit rules, availability, cost or latency preferences, or a learned model-selection policy. The supported policies depend on the product.
Model routing chooses which model should answer a request. Provider routing chooses where a selected model runs. A fallback can switch providers or models after a failure; that is different from choosing a model based on the task before the first call.
A gateway provides a common entry point for model API requests and may add authentication, logging, caching and usage controls. Routing is one capability a gateway can provide. A model-selection framework may instead run inside your application without being a full gateway.
No. Evaluate quality, total request cost and latency on your own workload. Include retries and fallback calls in the measurements, and check tool calling, structured output and data-handling requirements before switching models or providers.
An LLM router directs inference requests to models or providers. MCP Connect connects MCP servers and routes tool calls. An agent can use both: a model reasons about a task, then calls tools through MCP.
