Choose the model, control the provider, and plan for failures. Explore the infrastructure between your application and its model APIs.
These products have different scopes. This is a sourced starting point, not a performance ranking.
Hosted gatewayAccess models through one API and control which inference providers serve a request. Provider preferences and fallbacks help manage availability.
Provider selectionFallbacks
Before you integrateCheck provider-level data policies and model parameter support when setting routing preferences.
Self-hosted gatewayRun a proxy or use a Python router to distribute requests across model deployments, with load balancing, retries, cooldowns and fallbacks.
Load balancingFallbacksPython
Before you integratePlan deployment, provider credentials and shared usage tracking when running the proxy across multiple instances.
AI gatewayUse a unified API with conditional routing, provider and model fallbacks, caching and guardrails.
Conditional routingFallbacksGuardrails
Before you integrateReview the configuration and deployment options that apply to your team's governance requirements.
Hosted gatewayRoute requests across providers and fallback models with visibility into routing attempts, latency, token usage and cost.
Provider selectionFallbacksObservability
Before you integrateSet provider preferences explicitly and inspect response metadata to confirm which provider served the request.
Hosted gatewayManage model requests with logging, caching and rate limits, plus fallback providers for errors or configured timeouts.
FallbacksObservabilityCaching
Before you integrateDefine error and timeout behavior, and verify the authentication requirements of the endpoint you use.
Model-selection framework
Serve and evaluate learned routers that choose between a stronger and a weaker model using a configurable cost-quality threshold.
Model selectionEvaluationPython
Before you integrateCalibrate thresholds on representative queries and evaluate the model pair you actually intend to deploy.
Hosted gatewayUse one inference API with model routing, budgets and data controls. Choose ordered failover, intelligent selection or a custom benchmark-weighted routing policy.
Model selectionFallbacksSpend controls
Before you integrateUse an inference key rather than a management key. Check model eligibility against vendor, region, credential and data-retention restrictions.
Hosted gatewayRoute model requests through an OpenAI-compatible endpoint with provider fallbacks and request observability.
FallbacksObservability
Before you integrateCheck provider routing, credential handling and logging settings for the models you plan to use.
Self-hosted gatewayRun an OpenAI-compatible gateway with provider failover, load balancing and a web interface for configuration.
Load balancingFallbacksCaching
Before you integratePlan provider credentials and gateway hosting. Check which governance capabilities require an enterprise deployment.
Self-hosted gatewayAn open-source gateway with routing, observability and evaluation. Its official repository is archived; assess maintenance before adopting it.
FallbacksEvaluationPython
Before you integrateThe official repository was archived on June 12, 2026. Review maintenance and dependency risks before deploying. A published Python package does not establish ongoing support.
Hosted gatewaySend model requests through an OpenAI-compatible endpoint with fallback, load-balancing and latency-based routing policies.
FallbacksLoad balancingObservability
Before you integrateUse a Requesty key and configure the gateway base URL. Confirm current platform fees and data-handling terms directly with the publisher.
LLM routing, explained.
What is an LLM router?
An LLM router selects the model or inference provider that receives a request. Routing may follow explicit rules, availability, cost or latency preferences, or a learned model-selection policy. The supported policies depend on the product.
How is model routing different from provider routing?
Model routing chooses which model should answer a request. Provider routing chooses where a selected model runs. A fallback can switch providers or models after a failure; that is different from choosing a model based on the task before the first call.
Is an AI gateway the same as an LLM router?
A gateway provides a common entry point for model API requests and may add authentication, logging, caching and usage controls. Routing is one capability a gateway can provide. A model-selection framework may instead run inside your application without being a full gateway.
Does routing guarantee lower cost or better answers?
No. Evaluate quality, total request cost and latency on your own workload. Include retries and fallback calls in the measurements, and check tool calling, structured output and data-handling requirements before switching models or providers.
How does an LLM router relate to MCP Connect?
An LLM router directs inference requests to models or providers. MCP Connect connects MCP servers and routes tool calls. An agent can use both: a model reasons about a task, then calls tools through MCP.