The first version of every chatbot sends the whole conversation to the model on every call. It works, until it doesn't:
- Every turn costs more than the last. Input tokens grow with the conversation. Turn 200 pays for turns 1 to 199 again.
- Every turn is slower. More input means more time before the first token.
- Eventually it does not fit. The request passes the model's context window and the call fails, or the oldest messages are cut without anyone choosing which.
- Quality drops before that. Models use long inputs unevenly; Chroma's test of 18 models found performance "grows increasingly unreliable as input length grows" (context rot).
previous_response_id in the Responses API hides the resending, but not the cost: OpenAI says "all previous input tokens for responses in the chain are billed as input tokens" (conversation state). See previous_response_id and the Conversations API.
This page covers the fixes in the order most people reach for them: count, trim, summarise, and finally stop sending the history yourself.
When does a chat history stop fitting?
When input plus output passes the model's context window. OpenAI's definition: the window is "the maximum number of tokens that can be used in a single request", and that number "includes input, output, and reasoning tokens" (conversation state).
Windows as OpenAI's model pages list them on 12 November 2026:
| Model | Context window | Max output |
|---|---|---|
| Current flagship models | 1.05M tokens (models) | Varies by model |
| GPT-5 | 400,000, with 272,000 max input (GPT-5) | 128,000 |
| GPT-4.1 | 1,047,576 (GPT-4.1) | 32,768 |
A large window delays the failure. It does not change the cost curve: at a million tokens of history, each call still sends a million tokens. If you are already seeing the error, OpenAI context_length_exceeded covers the messages and fixes in detail.
Context window vs max_output_tokens
They are different limits. The context window bounds input and output together. max_output_tokens (Responses) or max_completion_tokens (Chat Completions) caps only the reply. Raising the output cap on a request whose history already fills the window makes the problem worse, not better: the reply has to fit in what the history left.
Step 1: count tokens before you send
You cannot trim to a budget you do not measure. tiktoken's o200k_base is the encoding for current OpenAI models (tiktoken):
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
def tokens(messages: list[dict]) -> int:
# Content only; message framing, images and tool schemas add more.
return sum(len(enc.encode(str(m.get("content") or ""))) for m in messages)Treat the count as an estimate and leave a margin. Each response's usage gives the real input count; compare the two now and then.
Step 2: trim safely
Trimming is slicing the history before each call. Two rules keep the request valid:
- Keep the system (or developer) message. It is your instructions, and it belongs at the top of every request.
- Never separate a tool call from its result. Chat Completions needs a
toolmessage for every tool call in the assistant message before it; cut between them and the API rejects the request. Cutting at a user message is always safe.
def trim(messages: list[dict], budget: int) -> list[dict]:
head = [m for m in messages if m["role"] in ("system", "developer")]
body = [m for m in messages if m["role"] not in ("system", "developer")]
for i, m in enumerate(body):
if m["role"] == "user" and tokens(head + body[i:]) <= budget:
return head + body[i:]
# Not even the last turn fits: send from the newest user message.
last = max(i for i, m in enumerate(body) if m["role"] == "user")
return head + body[last:]type Msg = { role: string; content?: unknown };
export function trim(messages: Msg[], budget: number, count: (m: Msg[]) => number): Msg[] {
const head = messages.filter((m) => m.role === 'system' || m.role === 'developer');
const body = messages.filter((m) => m.role !== 'system' && m.role !== 'developer');
for (let i = 0; i < body.length; i += 1) {
if (body[i].role === 'user' && count([...head, ...body.slice(i)]) <= budget) {
return [...head, ...body.slice(i)];
}
}
const last = body.map((m) => m.role).lastIndexOf('user');
return [...head, ...body.slice(Math.max(last, 0))];
}Trim by tokens, not by message count. Ten messages can be 500 tokens or 50,000, depending on what the tools returned.
Step 3: summarise what you trim
Trimming forgets the start of the conversation: the user's goal, their name, the decision made on turn 3. A summary keeps some of it. Before dropping old turns, have a model rewrite them into a short block that stays near the top:
def fold(messages: list[dict], budget: int) -> list[dict]:
kept = trim(messages, budget - 2_000) # leave room for the summary
dropped = [m for m in messages if m not in kept]
if not dropped:
return kept
text = "\n".join(f"{m['role']}: {m.get('content') or ''}" for m in dropped)
summary = openai.chat.completions.create(
model="gpt-5",
messages=[
{"role": "system", "content": "Summarise. Keep names, ids, numbers and decisions verbatim."},
{"role": "user", "content": text},
],
).choices[0].message.content
head = [m for m in kept if m["role"] in ("system", "developer")]
body = [m for m in kept if m["role"] not in ("system", "developer")]
return head + [{"role": "system", "content": f"Earlier in this conversation: {summary}"}] + bodySummaries lose specifics, and each rewrite of a summary loses more. They also change the start of the prompt every time they are rewritten, which defeats OpenAI's prompt cache: reuse needs the rendered prefix to match exactly (prompt caching).
The usual fixes, and what each loses
| Fix | How it works | What it loses |
|---|---|---|
| Keep the last N messages | Slice the array before each call | Everything before the slice: names, decisions, the user's goal |
| Summarise old turns | A model rewrites history into a running summary | Detail, and the summary drifts as it is rewritten |
| Retrieve with embeddings | Store messages in a vector database, search on each turn | Order and recency; what ranks highest is not what matters now |
| Compaction | OpenAI replaces old context with an encrypted item past a threshold (compaction) | You cannot read it, and details do not come back on demand |
Each is a reasonable start. The trouble is that each solves one problem: size. An agent needs the request to fit and to keep what it learned and to recall the past when the user brings it up. Context window management compares the four approaches in more depth.
Step 4: send only what is new
With Threadless AI your agent sends only what changed since the last call: the user's new message, or the model's reply and your tool results. It gets back the full request.
// Turn 200: send one message, get back a request that fits.
const request = await memory.chat.decorate(memoryId, [{ role: 'user', content: text }], { model, tools });
const reply = await openai.chat.completions.create(request);
memory.chat.report(memoryId, reply);What comes back:
- The latest messages, word for word. The newest part of the conversation is sent as it happened.
- Facts from older messages stay in view. Older messages stop being sent word for word, but before they do, your model notes their key facts with the note tool. Those facts sit in a facts block near the top of the request, and newer facts replace older ones.
- Older messages that matter come back when the user mentions something old: the exact messages and facts, with their dates, placed just before their message.
Token use stays bounded: the request stops growing with the conversation. How it works shows a request at message 413.
What you keep
- Your loop. Tool calls, retries and streaming work as before.
decorateruns before every call, including each call inside a tool loop. - Your key. The request goes from your process to OpenAI. It never passes through us.
- Your fallback. If we cannot be reached, the SDK builds the request from a local tail of recent turns and your call goes ahead. See What happens when memory is down.
Questions people ask
Why does my chat get slower and more expensive every turn?
Because each call sends the whole history again. Input tokens, and the time to read them, grow with the conversation, so turn 200 pays for the 199 turns before it.
How many messages should I keep?
Budget by tokens, not messages: a message can be ten tokens or tens of thousands. Decide how much of the window the history may use, and trim to that at a user message.
Is summarising better than trimming?
It keeps more of the old conversation, at the cost of an extra model call, lost detail and drift. Trimming is exact about recent turns and forgets the rest. Many apps trim and keep a short summary of what was trimmed.
Does previous_response_id fix a long chat history?
It stops you resending the history, but not paying for it: every earlier input token in the chain is billed again on each call. It also does nothing about the window filling up.
How do I limit the history I send to the OpenAI API?
Keep your system message, count tokens, and send only the newest turns that fit your budget, cut at a user message so tool calls keep their results. Or send only what is new to a memory service that returns the request.
Will a model with a bigger context window fix it?
It postpones the error, not the cost. Every call still sends, pays for and waits for the whole history, and models use very long inputs less reliably than short ones.