Enterprise AI Gateway and Model Router Guide
What you’ll learn
- The different roles of AI gateways, model routers, usage records, and quality evaluation
- Which environments suit Cloudflare, Vercel, Portkey, LiteLLM, Envoy, and Kong
- How to choose between RouteLLM, vLLM Semantic Router, Amazon Bedrock, and Microsoft Foundry
- How Langfuse, Phoenix, and OpenTelemetry support usage tracking and quality checks
- Selection criteria and considerations for automation across team trials, enterprise-wide adoption, and regulated work
The different roles of AI gateways and model routers
An AI model is the core AI that produces answers or decisions; models differ in their strengths and costs. An application is the program or interface used to request a task, such as a document summary. An AI gateway (a shared entry point that applies usage rules and manages where applications send AI requests) has a different role from a model router (a system that selects an AI model for each request). Enterprise AI operations also need ways to record actual usage and evaluate answer quality. The cloud means computing and storage rented from external providers.
For example, an application that summarizes internal documents needs two separate decisions: whether a document may be sent to an external AI service, and which model should summarize it. After execution, the application records the model and cost, and the summary is checked for missing information. Choosing a destination and getting an answer suitable for business use require separate checks.
A gateway alone does not establish quality evaluation or allocate costs to business outcomes. A router does not handle every requirement for data classification, budgets, audit (checking from records whether agreed rules were followed), or stopping requests during failures.
This article separates usage rules, model selection, usage records, and quality evaluation to explain product selection and rollout order. By the end, you will be able to choose a combination that fits your operating team and identify the checks needed before production use.
Four responsibilities and their owners in enterprise AI operations
This article organizes AI operations into four layers: enforcement, selection, observation, and evaluation. A layer represents a responsibility; it does not mean that four separate products are required.
| Role | Responsibilities | Decision | Main options |
|---|---|---|---|
| Enforcement: applying usage rules | Identity and permission checks, data handling, usage limits, retries and blocking | Who may send which data to which destination? | Cloudflare, Portkey, LiteLLM, Envoy and Kong gateways; cloud-native controls |
| Selection: choosing models and destinations | Model and provider selection, alternatives during failures, limited trials of new routes, stronger models when quality is insufficient | Which model should perform this task? | Gateway routing rules, RouteLLM, vLLM Semantic Router, Bedrock, Microsoft Foundry |
| Observation: recording usage and costs | Execution records, usage, response time, errors, reuse of stored information, allocation of costs to departments | What processing took place, and who owns the cost? | OpenTelemetry, Langfuse, Phoenix, gateway records |
| Evaluation: checking answers and business results | Quality, safety, task completion, quality regressions, cost effectiveness | Does a cheaper model still meet the business acceptance criteria? | Langfuse, Phoenix, internal evaluation tools, cloud evaluation features |
Even when one product covers several roles, each needs an owner. This makes it clear who should decide what when a failure or quality problem occurs.
Security owns the conditions for permitted use. Platform operations owns request routes and service availability. FinOps (the practice of managing cloud costs and value across technology, finance, and business teams) owns cost allocation (assigning shared costs to departments or uses) and budgets. Business teams judge answer quality and work outcomes.
One person should not request, approve, implement, and evaluate an exception. This separation of duties (assigning tasks such as requesting and approving to different people) also helps reduce change-related mistakes outside regulated work.
AI gateway selection for existing infrastructure
Gateway selection depends on deployment time, self-hosting requirements, existing networks, regulation, and operations staff. For a company with a small operations team, a managed service operated by its provider and one evaluation platform can be a practical combination.
Self-hosting means deploying and operating the service yourself. Its cost includes maintaining stored records, keeping the service running through failures, regularly updating keys used to verify service access, responding to provider changes, and handling incidents outside working hours. License fees alone do not show the full cost.
Existing infrastructure includes application-development components, cloud computing and storage, and network paths for requests. An authentication key is information used to verify permission to access a service.
The following comparison helps narrow the candidates. Check responsibility for costs and incident response as well as feature availability.
| Option | Suitable environment | Main checks before adoption |
|---|---|---|
| Cloudflare AI Gateway | Cloudflare is a shared traffic path, and the team needs routing and budgets across multiple models | How spending is counted, rule limits, supported providers, data paths [1][2] |
| Vercel AI Gateway | Application development centers on Vercel or AI SDK (development components for calling AI from an application) | Direct provider charges through your own keys and the scope of budget controls [3][4] |
| Portkey | The team wants one place to manage AI destinations, costs, and usage rules | Plan features, conditions for reusing stored answers, data retention, self-hosting scope [5] |
| LiteLLM Proxy | A team using Python (a programming language) wants to operate connections to many models itself | Storage needed to record budget usage, conditions for allowing traffic during failures, free versus paid features, operations ownership [6] |
| Envoy AI Gateway | Kubernetes (a platform for deploying and running applications across servers) is standard and a network team manages traffic processing | Readiness for rapidly changing specifications and separate management screen and evaluation tools [7] |
| Kong AI Gateway | The team wants separate configuration management and traffic processing, or uses Kong already | AI Gateway 2.x versus Kong Gateway deployment, paid features, storage needed to select processing from the meaning of input, failure detection [8][21] |
| AWS and Azure native features | Data, user permissions, and audit should stay within the existing cloud | Available models and routing conditions, portability to other clouds, repeatable evaluation conditions [11][12] |
Check what budget controls actually cover. Cloudflare spend limits use estimated usage and delayed aggregation, so they do not replace a ledger or guarantee that charges stop at an exact amount.[2] In Vercel, costs paid directly to a provider through your own authentication key are outside Vercel’s budget controls.[4]
Kong AI Gateway 2.x and AI plugins on Kong Gateway do not simply share configuration and credentials. Check the mapping even when migrating from an existing Kong installation.[8][21] The former Envoy AI Gateway documentation now redirects to the Agent Router site, so confirm the project and version you intend to adopt.[7]
Model router options and conditions for use
Model routers select models for a task to balance quality and cost. They work alongside gateways that handle usage rules and audit, and should be applied to work for which the company has verified quality.
Comparison Scope and Self-Operation with RouteLLM and vLLM Semantic Router
RouteLLM is a research-oriented framework (software for building and testing implementations) for evaluating and serving routers between stronger and lower-cost models. On MT-Bench, a public benchmark (a comparison using the same evaluation tasks), it reports up to 85% cost reduction while retaining 95% of GPT-4 performance.[9]
Those results depend on the evaluation conditions. The same savings and quality are not guaranteed for a company’s own work.
vLLM Semantic Router is software with publicly available implementation that selects models, tools, and reasoning paths (the processing route used to produce an answer) from input content and policy. It uses the Apache-2.0 license, which permits use, modification, and redistribution under its terms.[10]
It suits teams building detailed selection rules, including for self-hosted models. It does not replace the full set of model-serving infrastructure, authentication, audit, and quality evaluation.
Supported Models and Evaluation Conditions in Bedrock and Foundry
Amazon Bedrock Intelligent Prompt Routing balances quality and cost for each request between two configured models in the same family. It is optimized for English inputs and cannot be customized using application-specific performance data. Because it requires a baseline model and another model, teams should first check that the language and evaluation conditions fit their use case.[11]
Microsoft Foundry model router offers Balanced (balancing quality and cost), Quality, and Cost modes, along with configurable model subsets. Its official guidance calls for representative workload evaluation across quality, cost, tail response time (the waiting time for the slowest portion of requests), and policy fit.[12][13]
Tools for usage records and quality evaluation
Records of the model used and its cost need to be combined with an assessment of whether the result is suitable for the task. Langfuse and Phoenix are candidates for observability (using execution records to understand what happened inside a system) and evaluation.
Langfuse includes records of the steps from receiving a request to returning its answer, prompts (instructions and information supplied to AI), evaluations, datasets (collections of tasks or data for evaluation), and experiments as core features of its publicly available software. Enterprise features add finer access controls, audit, retention management, SCIM (synchronizing accounts as employees join or leave), and server-side masking (hiding confidential information after it is received).[14]
Phoenix is an OpenTelemetry-based platform with a publicly available program for records of the steps from receiving a request to returning its answer, evaluations, datasets, experiments, and a playground (an interface for trying different prompts and models). It can run locally or in a self-hosted environment.[15]
OpenTelemetry is not an evaluation product with the same role as these tools. It provides common conventions that let different tools use the same records. Its generative AI attributes include input and output token counts (units used to split and measure text processed by AI), token counts for reusing cached input processing, and the tokens used while the AI works toward an answer.[16]
Recording the adopted specification version and custom attributes preserves the meaning of records when the conventions change. This gives the systems sending and receiving the data an agreed definition of what is recorded and in which format.
Recommended configurations at three governance levels
Governance (deciding who makes decisions, which AI usage rules apply, and how compliance is checked) differs between team trials, enterprise-wide use, and regulated work. This article groups those requirements into three levels.
Separate the environment used to test changes from the one used for real business work. Confirm permitted models, exception approvers, and how to restore settings.
| Level | Intended use | Recommended configuration | Conditions for production |
|---|---|---|---|
| 1. Team evaluation | One to three teams seeking usage visibility for low-impact work | Existing cloud or a managed Vercel, Cloudflare, or Portkey gateway + OpenTelemetry + Langfuse or Phoenix | User and use-case records, limits, execution records, and fixed evaluation tasks |
| 2. Shared enterprise platform | Multiple departments using multiple providers and allocating costs | Managed gateway, or LiteLLM, Envoy, Kong AI Gateway 2.x, or Kong Gateway AI plugins; shared sign-in, credential management, centralized records, and automated evaluation on changes | Automated development-production separation, model allowlists, departmental budgets, exception approval, and restoration of earlier settings |
| 3. Regulated or high-impact work | Confidential data or customer decisions requiring strict audit | Cloud in an approved region or a dedicated traffic-processing platform; policy enforcement, tamper-resistant audit records, and human approval | Evidence of data classification, retention and deletion, separation of duties, change review, and regular independent evaluation |
For a shared platform, SSO (signing in to multiple services with a common account) and secret management (storing credentials and releasing them only to authorized processes) manage users and permissions. Evaluation CI (automatically running evaluations after configuration or code changes) detects quality regressions.
The NIST AI RMF (a framework for managing AI risks) organizes continuing work into Govern, Map, Measure, and Manage: establishing governance, understanding use and risks, evaluating them, and responding. ISO/IEC 42001 (an international standard for organizational AI management systems) defines continual improvement.[17][18]
These frameworks require more than checking product features. They inform ownership, evaluation methods, exceptions, evidence, and review cycles.
For cost allocation, FinOps practices support recording the user, application, department, development or production environment, and workflow (a sequence of business tasks) when a request is made. Rules for dividing shared costs between departments also need to be explicit.[19]
Visibility, usage rules, and evaluation as prerequisites for automated model selection
Automated model selection needs records that show what happened in each workflow, rules for permitted data and models, and quality evaluation for each task. Check that the team can identify the cause of a cost or quality change and restore the previous configuration.
| Prerequisite | What to compare or record |
|---|---|
| Usage visibility | Purpose, department, model, input and output usage, cost, response time, and failure rate |
| Usage rules | Permitted data, processing region, budget, operation permissions, and stop or fallback behavior |
| Quality evaluation | Pass rate on the same business cases, serious errors, human correction time, and regressions after changes |
This table is an operational framework. Associate records with departments and purposes without indiscriminately storing full confidential inputs.[16][19]
Some caches reuse processing for a shared part of the input; others reuse the answer itself. Requirements and benefits depend on the model, provider, and caching method.[20][22] Compare costs under matching cache conditions, including retries and human correction time. When reusing answers, test whether saved answers could be used for requests with different assumptions or permissions.
Work with stable quality after shortening input and correctable errors is a candidate for automated routing. Do not switch high-risk decisions or workflows with contractual model restrictions solely for potential savings. Retain the tested version and configuration, and define when quality loss or an outage should trigger a rollback.
Summary: A fit for the operating team and gradual automation after evaluation
Selecting an enterprise AI platform means assigning owners for usage rules, model selection, records, and quality evaluation, then choosing tools that fit the existing environment and staff. Checking quality, slow responses, failure rates, and the ability to restore earlier settings before expanding use brings cost reduction and business acceptance into the same decision.
The fit of automation depends on traceable usage and evaluation for each type of work. Tasks requiring reproducibility with a specific model are better suited to fixed routes and approval for model changes.
This article is a general information summary and is not legal or security advice. Confirm practical decisions with qualified specialists in your organization.
References
- Cloudflare, Dynamic routing, AI Gateway Docs
- Cloudflare, Spend limits, AI Gateway Docs
- Vercel, Provider Options, AI Gateway Docs
- Vercel, Budgets, AI Gateway Docs
- Portkey, AI Gateway, Documentation
- LiteLLM, LiteLLM Proxy Server, Documentation
- Envoy AI Gateway, Data Plane, Documentation
- Kong, AI Gateway architecture
- LMSYS Org, RouteLLM, GitHub
- vLLM Project, vLLM Semantic Router, GitHub
- Amazon Web Services, Use intelligent prompt routing with Amazon Bedrock, Documentation
- Microsoft, How to use model router for Microsoft Foundry Models, Microsoft Learn
- Microsoft, Evaluate model router for your workload, Microsoft Learn
- Langfuse, Self-hosting and product editions, Documentation
- Arize AI, Phoenix, GitHub
- OpenTelemetry, Generative AI Attributes, Semantic Conventions
- National Institute of Standards and Technology, AI RMF Core, AI Risk Management Framework
- International Organization for Standardization, ISO/IEC 42001:2023, December 2023
- FinOps Foundation, Capability: Allocation, FinOps Framework
- Amazon Web Services, Prompt caching for faster model inference, Amazon Bedrock User Guide
- Kong, Configure Kong AI Gateway on-prem
- Cloudflare, Caching, Cloudflare AI Gateway Docs
For current features, prices, licenses, and regional availability, check each service’s official website and documentation.