Enterprise Token and Model Management: Lessons from Uber and Spotify
What you’ll learn
- Centralized AI model management and evaluation on real tasks, illustrated by Uber
- Separating bulk reads and selecting the information an AI receives, illustrated by Spotify
- Common patterns across Uber, Spotify, and Grab, and six layers of enterprise operations
- Why cost per completed task that meets quality requirements matters more than token price alone
How workflows and context shape generative AI costs
Reducing generative AI costs requires reviewing how work gets done, as well as model prices. A cheaper model can make a task more expensive overall if incorrect answers and repeated attempts increase.
Prompt length (the length of the instructions sent to an AI) is only one cost driver. Repeated searches, descriptions of unused tools sent on every turn, and retries after failures all consume tokens (units of information processed by an AI). Producing more output without evaluating its quality also leaves teams unable to tell whether the spending delivered useful results.
Teams therefore need to match both the model and its context (instructions, documents, and conversation history needed for a decision) to the task. This article compares public examples from Uber, Spotify, and Grab to explain how to manage cost and quality together, including criteria for adoption and checks after implementation.
Here, tokens refer to information handled as input, output, reasoning, or cached content. Reasoning is the model’s internal processing toward an answer; caching is a way to reuse information already processed. These are different from access tokens used for authentication. Each company’s reported reductions also reflect its own environment and measurement conditions. The same results may not apply elsewhere.
Uber: Centralized AI access and evaluation on real tasks
Usage, cost, and data controls through a central gateway
Uber uses a generative AI gateway (a shared entry point that receives and manages access to AI). It provides an OpenAI-compatible interface (a common format for calling models) for both external providers and internal models.
The gateway brings several controls together.[1]
- Authentication and authorization (checking who a user is and what they may access)
- Personal-data redaction (hiding information that could identify someone)
- Usage records and audits
- Cost attribution (assigning spending to the teams that incurred it) and budgeting
Uber’s 2024 article reported nearly 30 teams using the gateway, with 16 million queries per month and peak throughput of 25 queries per second.[1]
The gateway does more than standardize API formats (the ways software calls another system’s functions). Access is granted after reviewing the use case and data handling. Records show which team sent how much information to which model through which route.
The controls themselves also need evaluation. Redacting personal data can improve protection, but it may slow responses or remove information needed for a good answer.[1]
Six drivers of rising costs
Uber’s 2026 account of AI-assisted software development breaks costs into six factors.[2]
Users × sessions per user × turns per session × requests per turn × tokens per request × price per token
A session is a connected period of work or conversation. A request is an individual processing call sent to a model.
This breakdown helps distinguish a bill rising because of more users, more tool calls, more information per request, or a higher model price.
Once the cause is clear, restricting usage is not the only option. Teams can cache repeatedly read information, summarize long conversation histories, load tool descriptions only when needed, or divide work into smaller tasks for cheaper models.
Cost per completed task that meets quality requirements
Uber does not select models solely by name or public benchmarks (tests that compare performance on shared tasks). It compares them in a model-independent test environment using tasks that resemble the intended work.[2]
The comparison covers quality, completion rate, reliability, and cost. Among options that maintain quality, the aim is to select a model with a lower cost per completed task.
The article also describes using cheaper small models by default for supporting tasks, while the primary model divides the work and reviews results. Different models can take different roles in the same workflow.[2]
Loading only the tools a task needs
Tool definitions (descriptions of what tools do and how to use them) also affect cost. Uber reports that definitions for roughly 100 tools can consume 50,000 to 70,000 tokens.[2]
The article describes searching for and loading only the tools needed, as well as execution methods that keep raw tool output out of conversation history.
For some workloads, reviewing what the model needs to see can reduce costs more effectively than switching models.
Spotify: Separating bulk reads and supplying relevant company information
Bulk file reading through Shunt
Spotify’s public Shunt plugin (an extension that adds functionality) is designed for coding agents (AI systems that investigate and edit code). It delegates bulk file reading to another AI system.[3]
The process has three steps.
- A hook (a mechanism that inserts processing before or after another operation) blocks reads exceeding 350 lines.
- A dedicated script (a program for a specific operation) reads the target content.
- Only the relevant, organized information is returned to the original agent.
Suitable tasks include summarizing large files, investigating multiple files, and extracting tests or dependencies (relationships in which one program relies on another).
The public README (a document describing usage and specifications) reports a comparison using a 162,000-line Java monorepo (one code repository containing multiple projects). Across three bulk-read tasks, token use by the original agent fell by 82% to 94%, averaging 90%.[3]
These figures describe token reductions for the original agent. They should be distinguished from total cost reductions that also include the other AI system.
Shunt explicitly excludes debugging (finding and fixing defects), small-file operations, architectural judgment, and interactive editing from delegation. Delegating code generation is advisory rather than mandatory.[3]
The target is work involving large amounts of I/O (reading and writing data) whose results can be returned in a short, organized form.
Company information for more focused searches through Portal
Giving an AI less information is not always better. Supplying the right information upfront can reduce investigation and trial and error.
Spotify Portal connects AI to a service catalog (a list of internal services), service owners, compliance with internal standards, and internal tools.
Spotify published one controlled comparison using the same model, codebase (the collection of code being worked on), and tasks. Providing organizational context (company-specific information needed for decisions, such as ownership and internal rules) upfront produced these results.[4]
| Metric | Reported result |
|---|---|
| Token cost | 32% lower |
| Time to completion | 65% shorter |
| Correct answers | Increased from 4 of 10 to 8 of 10 |
This is a single internal case, so the results cannot be assumed to apply to other work. It does suggest that information which reduces searching and incorrect attempts may lower total task costs even if it adds tokens upfront.
Model switching with preserved work context through Xirp
Xirp is described as an agent environment that is independent of any single AI vendor and preserves work context when switching models or tools.
An August 2026 article reported more than 36,000 sessions and described switching to models with suitable price-performance, including self-hosted models (models run in infrastructure provided by the organization itself).[5]
Together, these examples suggest two areas of focus beyond model prices.
- Delegate bulk reading, searching, and formatting so the original agent’s conversation history does not accumulate unnecessary information.
- Supply ownership, internal standards, and dependency information when it can shorten investigation.
Common patterns across three companies and a six-layer operating model
Grab’s AI Gateway and a comparison of the three companies
Grab’s AI Gateway also centralizes authentication for multiple external and internal models. It shares reserved processing capacity while recording requests, responses, token counts, models, and processing routes.[6]
It separates authentication keys (credentials required for access) for exploration and production, combining use-case reviews, cost attribution, usage limits, and routing based on current conditions. Grab’s 2025 article reported applications from more than 3,000 employees and over 300 use cases (specific ways of applying the technology).[6]
The companies differ in scale and implementation, but comparing what they manage and how they make decisions reveals common patterns.
| Area | Uber | Spotify | Grab |
|---|---|---|---|
| Main management mechanism | Company-wide gateway | Agent hooks and an internal portal | Company-wide gateway |
| Main improvement targets | Models, tools, conversation history, caching | Bulk I/O, investigation, organizational information | Processing capacity, routes, budgets, use cases |
| Work allocation | Cheaper small models for supporting tasks | Delegation of bulk reads | Routes selected by use case and capacity |
| Main evaluation measures | Task completion, quality, reliability, cost | Tokens, time, and correctness by task | Usage and cost by user, model, and route |
| Support for operational rules | Reviews, personal-data redaction, audits, budgets | Ownership, internal standards, tool information | Reviews, environment separation, cost attribution, limits |
The implications for an organization’s own design can be summarized as follows.
| Area | Design implication |
|---|---|
| Management mechanism | Apply common rules centrally while dividing work into suitable tasks locally |
| Improvement targets | Review information volume, investigation, and routes alongside token price |
| Work allocation | Use a cheaper default and define when to switch to a higher-performance model |
| Evaluation measures | Assess cost against completion at the required quality |
| Operational rules | Use common workflow identifiers for access decisions and cost management |
Six layers from work to outcomes
Based on these examples, this article proposes six layers for enterprise token and model management. The companies do not necessarily use a framework with these names.
| Layer | Decisions | Minimum records |
|---|---|---|
| 1. Work | Whether the task is summarization, extraction, generation, reasoning, search, or execution | Use case, team, deliverable, business impact |
| 2. Context | Required instructions, information to retrieve, tools to load on demand, information to cache | Prompt version, information source, tool name, cache writes and reads |
| 3. Model | Default model, escalation conditions, exceptions requiring a fixed model | Requested model, executed model, routing reason |
| 4. Controls | Authentication, data classification, usage limits, stop conditions, retries, human approval | User, department, environment, policy decision, exceptions |
| 5. Observation | Input, output, reasoning, and cached tokens; response time, failures, cost | ID for tracing the processing sequence, price version, status, estimated and final costs |
| 6. Outcomes | Quality, completion rate, rework count, completion time | Evaluation-set version, acceptance criteria, cost per completed workflow |
Fixed, default, and escalation models
Three model roles can make operations easier to manage.
| Role | Main use |
|---|---|
| Fixed model | Work where regulatory requirements, reproducibility, or certified behavior take priority |
| Default model | Routine standard work |
| Escalation model | A higher-performance model used when inadequate quality or task complexity is detected |
Even when a router (a mechanism that selects a model or processing route) chooses automatically, record the requested model, the model actually used, and the reason for the selection. These records make the decision traceable afterward.
Monthly spending and cost per workflow
Budgeting should track input, output, reasoning, and cache costs alongside the monthly total. It should also compare cost at several levels.
- Per 1,000 requests
- Per user
- Per session
- Per completed workflow
Cost per completed workflow deserves particular attention. If token prices fall but this measure does not improve, additional rework may be consuming the savings.
Attach cost-attribution metadata (additional information used to classify and manage processing) when sending each request. Record the application, department, environment, and workflow (the sequence of steps in a business process), as well as the user.
This makes access decisions, budgets, and outcomes easier to track against the same unit of work than trying to infer classifications later from logs (records of processing activity).
A starting point: Reviewing the 20 highest-cost workflows
Begin the analysis by identifying the 20 highest-cost workflows and classifying them as bulk reading, repeated search, generation, complex reasoning, or tool execution.
For each workflow, bring four items together in one table.
- Information the AI must receive
- The model currently used
- Conditions for treating the task as complete
- The number of retries or rework attempts
This helps reveal whether bulk I/O can be separated, whether unused tool definitions are being supplied, and why searches go in the wrong direction. Workflow improvements can be investigated before negotiating model prices.
If a workflow lacks representative quality evaluation tasks or clear completion criteria, automatic routing (automatic model or processing-route selection) should not become the production default. Use the central gateway to make usage visible and set limits first. Keep model changes within the scope of human approval.
The enterprise AI gateway and model router selection guide covers specific products and OSS (software whose source code is available for use and modification under its license).
Summary: Cost per completed task at the required quality
The three companies’ examples suggest that managing generative AI cost and quality requires designing how work is divided and information is supplied, alongside choosing models. Both cheaper models and reductions in unnecessary context should be assessed against the cost of finishing the task, including quality and rework.
An organization can begin by recording the required information, current model, completion criteria, and rework count for its 20 highest-cost workflows. Where representative evaluation tasks and completion criteria are not available, usage visibility and limits come first, with model changes subject to human approval.
This article is a general information summary and is not legal or security advice. Confirm practical decisions with qualified specialists in your organization.
References
- Uber Engineering, Uber’s Gen AI Gateway, July 11, 2024
- Uber Engineering, Running a Software Factory Efficiently at Uber Scale, August 27, 2026
- Spotify, Shunt: Stop wasting Claude Code tokens, GitHub
- Spotify Portal, The Hidden Tax on Your AI Agents, August 28, 2026 (updated September 3, 2026)
- Spotify Portal, What we’ve learned scaling AI coding agents at Spotify, August 10, 2026
- Grab Engineering, Grab’s AI Gateway, February 19, 2025
For current features, prices, and published figures, check each company’s official website and documentation.