# Context window management for LLM agents: 4 strategies

> How to keep a long agent conversation inside the context window: truncation, summaries, retrieval and keeping what matters, with token budgets, caching, costs.

A context window is a budget. Instructions, tools, history, retrieved material and the reply all have to fit, on every call. Context window management is deciding what gets the space when they don't.

There are four approaches in common use: truncation, summarisation, retrieval, and keeping what matters while bringing back the rest. Most production agents end up combining two or three. This page compares them on what each keeps, what each loses and what each costs, then covers the parts most guides skip: the token budget, prompt caching, tool output and how to tell whether it is working.

## Set a token budget first

The window covers more than the history. OpenAI defines it as the maximum number of tokens in a single request, and that number "includes input, output, and reasoning tokens" ([conversation state](https://developers.openai.com/api/docs/guides/conversation-state)). A request that fills the window with input leaves nothing for the answer.

Split the window before you decide what to cut. An example for a 200k-token window:

| Part | Example share | Notes |
|---|---|---|
| Instructions | 2-5% | Fixed. Measure it once per version. |
| Tool definitions | 1-5% | Fixed per agent; grows with every tool you add. |
| Retrieved documents (RAG) | 0-15% | Cap it, or it crowds out the conversation. |
| Conversation history | 50-70% | The part you manage. |
| Reply and reasoning | 15-25% | Headroom. Reasoning models need more. |

The fixed parts come off the top; the history gets what is left. Count rather than guess. OpenAI's tokeniser for current models is `o200k_base` in [tiktoken](https://github.com/openai/tiktoken):

```python
import tiktoken

enc = tiktoken.get_encoding("o200k_base")

def tokens(messages: list[dict]) -> int:
    # Content only; each message also carries a few tokens of framing.
    return sum(len(enc.encode(str(m.get("content") or ""))) for m in messages)
```

```ts
import { getEncoding } from 'js-tiktoken';

const enc = getEncoding('o200k_base');
const tokens = (messages: Array<{ content?: unknown }>) =>
  messages.reduce((n, m) => n + enc.encode(String(m.content ?? '')).length, 0);
```

These are estimates: message framing, images and tool schemas add tokens a plain count misses. The `usage` field on each response is the provider's real count, so correct your estimate with it. Threadless AI uses that count too, which is why `report` should carry the response's `usage` ([Models and formats](https://threadlessai.com/docs/models-and-formats#request-size)).

## 1. Truncation

Keep the newest messages and drop the oldest, by count or by tokens.

- **Keeps:** the recent turns, exactly.
- **Loses:** everything before the cut, including the reason the conversation started.
- **Watch for:** cutting between a tool call and its output, which the API rejects. Cut at a user message, and keep the system message.
- **Cost:** none to run, a few lines to build.

```python
def truncate(messages: list[dict], budget: int) -> list[dict]:
    system = [m for m in messages if m["role"] in ("system", "developer")]
    rest = [m for m in messages if m["role"] not in ("system", "developer")]
    # Candidate cut points: user messages, so a tool call never loses its output.
    for i, m in enumerate(rest):
        if m["role"] == "user" and tokens(system + rest[i:]) <= budget:
            return system + rest[i:]
    last_user = max(i for i, m in enumerate(rest) if m["role"] == "user")
    return system + rest[last_user:]
```

Truncation is the right fallback. It is what the Threadless AI SDK uses when our service cannot be reached: a local tail of recent turns, cut at a user message ([Failure behaviour](https://threadlessai.com/docs/failure-behaviour)).

## 2. Summarisation

When the history grows past a limit, a model rewrites the oldest part into a summary that stays at the top.

- **Keeps:** the gist.
- **Loses:** specifics (ids, numbers, names), and accuracy over time as the summary is summarised again.
- **Watch for:** drift. Each rewrite is a chance to lose or invent a detail.
- **Cost:** an extra model call per summary, and tuning of what the summary should keep.

```python
def summarise_old(messages: list[dict], keep_last: int = 20) -> list[dict]:
    system, old, recent = messages[:1], messages[1:-keep_last], messages[-keep_last:]
    if not old:
        return messages
    summary = openai.chat.completions.create(
        model="gpt-5",
        messages=[
            {"role": "system", "content": "Summarise this conversation. Keep names, ids, numbers and decisions verbatim."},
            {"role": "user", "content": "\n".join(f"{m['role']}: {m.get('content') or ''}" for m in old)},
        ],
    ).choices[0].message.content
    return system + [{"role": "system", "content": f"Earlier in this conversation: {summary}"}] + recent
```

In real code, move `keep_last` to a user message, as in the truncation example. OpenAI's compaction is a managed variant of this idea for the Responses API: past a `compact_threshold`, older context is replaced by an encrypted item that is "opaque and not intended to be human-interpretable" ([compaction](https://developers.openai.com/api/docs/guides/compaction)). See [OpenAI compaction vs a memory layer](https://threadlessai.com/guides/openai-compaction-vs-memory).

## 3. Retrieval

Store past messages (or facts drawn from them) with embeddings; on each turn, search for what relates to the new message and put the results in the prompt.

- **Keeps:** anything, in principle.
- **Loses:** order and time. A similarity search ranks a message from five minutes ago and one from five weeks ago the same way.
- **Watch for:** retrieval that is relevant to the words but not to the moment, and prompts that fill up with near-duplicates.
- **Cost:** a vector store, an embedding pipeline, a retriever, and code that decides where results go.

This is what most memory stores provide. You still decide what fits the window. See [Agent memory vs RAG](https://threadlessai.com/guides/agent-memory-vs-rag) for where retrieval stops being memory.

## 4. Keep what matters, bring back the rest

Send the latest messages word for word. Before older messages stop being sent word for word, have the model note their key facts, and keep those facts in view. When the conversation turns back to something older, bring the exact older messages back, with their dates.

- **Keeps:** the latest messages, the facts of everything older, and the older messages that matter to what was just asked.
- **Loses:** older messages from the request, word for word. They stay stored and come back when the conversation turns to them.
- **Watch for:** facts written badly. The model that was in the conversation writes the best ones, so we ask it to.
- **Cost:** a lot to build. Threadless AI runs it as a service; you make three calls.

## Side by side

| | Truncation | Summaries | Retrieval | Keep what matters |
|---|---|---|---|---|
| Fits the window every call | Yes | Yes | Only if you cap it | Yes |
| Keeps old specifics | No | Partly | Yes, if found | As facts, plus recall |
| Knows what is current | By position only | No | No | Yes: newer facts replace older ones |
| Brings back old details when asked | No | No | Yes, if found | Yes, with their dates |
| Readable state | Yes | Yes | Yes | Yes: facts and messages, exportable |
| You build and tune | Little | Some | A lot | Nothing, as a service |

## Why doesn't a bigger context window fix it?

A bigger window moves the wall; it does not remove the problem.

- **Models use long inputs unevenly.** In *Lost in the Middle*, performance "is often highest when relevant information occurs at the beginning or end of the input context" and drops when it sits in the middle ([Liu et al.](https://arxiv.org/abs/2307.03172)). Chroma tested 18 models and found that "performance grows increasingly unreliable as input length grows" ([context rot](https://www.trychroma.com/research/context-rot)).
- **Every call pays for everything in it.** Input tokens are billed on every call, and more input means a longer wait for the first token.
- **The conversation keeps going.** A window twice the size reaches its end in twice the turns.

The aim is not the largest request that fits. It is the smallest request that holds what the model needs now.

## How does context management interact with prompt caching?

OpenAI caches the start of a prompt, by default on supported models. Cache reuse needs the rendered prefix to match exactly, and the guide's advice is to "put stable developer instructions and shared reference material first"; cached input is billed at a discount ([prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching)).

That changes which strategy is cheap:

- **A sliding window that drops one old message per turn** changes the prefix on every call, so the cache never holds past your instructions.
- **Summaries rewritten every turn** do the same.
- **Cutting rarely and in large steps** keeps the prefix stable for many calls between cuts.

## Tool outputs and agent loops

In agents, tool results are usually the largest thing in the history: a file read, a search result page, a database dump. They also go stale fastest.

- **Keep the call and its output together.** Chat Completions needs a `tool` message for every tool call; dropping one half breaks the request.
- **Treat old tool output as the first thing to go.** The model rarely needs the raw output of a step it finished twenty turns ago; it needs what it learned from it.
- **Manage context inside the loop, not only per user turn.** A tool loop can add tens of thousands of tokens before the user speaks again. With Threadless AI, `decorate` runs before every model call, including each call inside a tool loop ([The three calls](https://threadlessai.com/docs/calls)).

## How to tell it is working

- **Tokens per call over time.** It should level off, not grow with the conversation.
- **Context-length errors.** These should be zero. See [OpenAI context_length_exceeded](https://threadlessai.com/guides/openai-context-length-exceeded).
- **What was dropped.** Log which messages left the request on each cut, so a bad answer can be traced to a missing fact.
- **Recall misses.** Keep a few test conversations where a detail from early on is asked about much later, and check the answer.
- **Cache hit rate.** OpenAI's caching guide suggests tracking `usage.input_tokens_details.cached_tokens` ([prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching)); a falling share means your prefix keeps changing.

## Choosing

- A short-lived assistant with conversations under the window: send everything, or truncate.
- A support bot where every conversation is one issue: truncation plus a short summary is often enough.
- A question-answering system over documents: retrieval, which is a different problem from conversation memory. See [Agent memory vs RAG](https://threadlessai.com/guides/agent-memory-vs-rag).
- An agent that works with the same person for weeks: keep what matters and bring back the rest, so the context window always holds what matters now. That is [infinite context](https://threadlessai.com/guides/infinite-context), and the wider topic is [AI agent memory](https://threadlessai.com/guides/ai-agent-memory).

## Questions people ask

### What is context window management?

Deciding what goes into each model call when everything you could send does not fit. It covers the token budget, what history to keep, what to drop or summarise, and what to bring back from storage.

### How do I prevent context length exceeded errors?

Count tokens before the call, leave headroom for the reply, and cut the history when it passes your budget, at a user message so tool calls keep their outputs. A managed approach such as Threadless AI keeps every request sized for a 200k-token context window for you.

### Sliding window or summarisation?

A sliding window is simpler and exact about recent turns but loses everything older. Summarisation keeps the gist of older turns but loses detail and drifts as it is rewritten. Many agents use both: recent turns verbatim, a summary of the rest.

### Does prompt caching change the strategy?

Yes. Caching rewards a prompt whose start stays the same. Strategies that cut a little every turn change the prefix every turn; cutting rarely and in larger steps keeps the cache useful.

### When should I use RAG instead of a bigger window?

When the material is a body of documents that exists before the conversation and is too large to send. RAG retrieves the passages a question needs; a bigger window would pay to send all of it on every call.

### How much headroom should I leave for the reply?

Enough for the longest reply you expect plus the model's reasoning, since output and reasoning tokens both count against the window. Check your own responses' `usage` to see how much reasoning your model spends on a typical turn.

## Related

- [Why agents should forget](https://threadlessai.com/guides/why-agents-should-forget)
- [How to stop resending the whole chat history](https://threadlessai.com/guides/stop-resending-chat-history)
- [AI agent memory, explained](https://threadlessai.com/guides/ai-agent-memory)
- [How it works](https://threadlessai.com/how-it-works)