What Is Prompt Caching?
Prompt caching reuses computation from an earlier request when a later request begins with the same prompt prefix. Its main purpose is to reduce the time and cost of repeatedly processing a long shared input from the beginning.
It is different from a response cache, which stores a finished answer, and from memory, which carries user information into later conversations.
What Gets Reused
An LLM splits input into tokens and computes how each token relates to what came before it as it generates an answer. Prompt caching stores and reuses this computed state for an unchanged prefix. OpenAI describes that state as key-value cache tensors.[1]
Request 1: [stable instructions + shared document] + [question A]
└──────── compute and store ──────────┘
Request 2: [stable instructions + shared document] + [question B]
└──────── reuse stored state ─────────┘ └ process anew ┘The cache reuses prefix computation, not an answer. The answer to question B is generated anew, so a cache hit does not guarantee identical output.[1]
What Produces a Cache Hit
The basic requirement is a match from the beginning of the input through a sufficiently long prefix. Inputs are therefore easier to reuse when arranged in this order:
- Tool definitions and stable rules
- Long-lived documents and examples
- Conversation history
- Request-specific questions or data
Changing instructions, tool definitions, images, documents, or their order near the front may prevent later content from being reused. Retention periods and explicit controls differ by service, so check the current requirements for the model and API in use.[1][2]
Claude and OpenAI API Examples
| Service | How the mechanism appears | Example control or measurement |
|---|---|---|
| Claude API | Caches a shared prompt prefix and reuses it in later requests | Set boundaries or retention with cache_control, then inspect cache-creation and cache-read token usage [2] |
| OpenAI API | Automatically reuses processing for long shared prefixes on supported models | Inspect values such as cached_tokens in the response [1] |
In the Claude API, cacheable content is assembled in the order tools, system messages, then regular messages. The OpenAI API also recommends putting stable content first and variable content later.[1][2]
How to Think About ChatGPT and Codex
ChatGPT projects and memory can supply related sources, instructions, preferences, and other useful information to future conversations.[3][4] Those features govern what information enters the context. They do not serve the same purpose as prompt caching, which reuses input-processing computation.
In Codex, repository instructions and conversation history may enter context repeatedly. When a developer uses the OpenAI API directly, a stable prefix like this may qualify for prompt caching. However, visible projects, memories, and histories in ChatGPT or Codex do not reveal which internal API cache, if any, was used.
How Related Mechanisms Differ
| Mechanism | What it handles | Main purpose | Generates a new answer? |
|---|---|---|---|
| Prompt cache | Computed state for the same input prefix | Reduce latency and input-processing cost | Yes |
| Response cache | A finished answer | Return the same result quickly for the same question | Usually no |
| Memory | Preferences, facts, and earlier highlights | Use information across conversations | Yes |
| Context compaction | Task state retained from a long history | Make the active context smaller | Yes |
The key distinction is that prompt caching does not reduce context size. Even when an input prefix is processed faster, that information still belongs to the context handled by the model. Shortening a long conversation requires context compaction.
Good Fits and Limits
Prompt caching works best when requests repeatedly share a long prefix:
- Agents that use the same system instructions and tool set
- Multiple questions about the same long policy or manual
- Batch processing that changes only the data after a prompt with many examples
- Repeated development tasks grounded in the same codebase description
It offers less benefit when every request changes from the beginning, when the reusable prefix is short, or when documents change frequently. A cache also does not guarantee correctness. Quality evaluation, freshness controls, and authorization still need separate designs.
Summary
- Prompt caching reuses computed state for the same prompt prefix
- Stable instructions and documents belong first; request-specific questions belong later
- A cache hit still generates a new answer and does not guarantee identical output
- ChatGPT projects and memory supply information; their purpose is different from prompt caching
- Caching does not shorten context. Context compaction organizes long histories
References
- OpenAI, Prompt caching
- Anthropic, Prompt caching
- OpenAI, Projects and chats
- OpenAI, Memories
Freshness note: Supported models, retention periods, and configuration options for caching can change. Check each service’s current specification before implementation.