The obvious goal for agent memory is to forget nothing: send the model every message, every time. It is the wrong goal. A model given the whole history has to work out, on every call, which parts still matter. It does that badly, it does it slowly, and you pay for every token it reads.
People do not remember that way. We forget most of what happened, keep what we learned from it, and get the details back when something reminds us. This page covers what keeping everything costs, the evidence that a smaller, well-chosen context does better, and what an agent should keep instead.
What sending everything costs
Money. Each call is billed for every input token. If every call resends the whole conversation, input tokens grow with every turn, and the total cost of a conversation grows faster than its length: turn 300 pays for the 299 turns before it, again.
Time. The model has to read the whole request before it answers. A long history means a slower first token on every call, including the ones that only need the last few messages.
Answers. Models do not use a long context evenly. In the "lost in the middle" study, models answered best when the information they needed sat at the start or the end of the prompt, and noticeably worse when it sat in the middle of a long one (Liu et al., 2023). A conversation's history is mostly middle. The more old, settled detail it carries, the easier it is for the one line that matters to be missed.
The end of the window. Eventually the history does not fit at all, and the call fails with a context-length error. See OpenAI context length exceeded.
Does a smaller context make agents better?
There is direct evidence that keeping everything hurts. Xiong et al. compared an agent memory that adds every experience with one that adds only those passing a quality check, across four agents (How Memory Management Impacts LLM Agents):
| Agent (task) | Add everything | Selective addition |
|---|---|---|
| EHRAgents (medical records QA) | 13.05% accuracy, 2,411 records | 38.50%, 1,012 records |
| AgentDriver (driving) | 32.32% success, 2,125 records | 51.00%, 1,178 records |
| RegAgent (regression) | 55.48% success, 4,100 records | 70.95%, 2,938 records |
| CIC-IoT Agent (network security) | 59.90% accuracy, 1,050 records | 85.40%, 904 records |
Adding deletion on top shrank memory further; for EHRAgents, it reached 42.34% with 248 records. The authors' summary: "The results highlight the necessity of selection for effective memory addition."
Two cautions. These agents store past task experiences, not the turns of one conversation, so the numbers do not transfer directly. The lesson that carries over is the one in the title: an agent working from less, chosen well, did better than one working from everything.
The usual fixes, and what they lose
| Approach | What the model sees | What goes wrong |
|---|---|---|
| Send everything | Every message | Cost and latency grow every turn; answers get worse; the window runs out |
| Sliding window | The last N messages | The start of the conversation is gone: the install date, the name, the decision |
| Summarise the old part | A summary plus recent messages | Summaries drift and drop the exact detail you need later |
| Keep what matters, bring back the rest | Recent messages, kept facts, and older messages that matter now | Needs a memory layer, which is what this page is about |
What an agent should keep
Forgetting well is not deleting. It is choosing what the model sees on this call:
- The latest messages, word for word. What just happened is what the model most needs to see exactly.
- The facts from older messages. The install is on the 14th; Sam is the contractor; layout two was chosen. They stay in view long after the messages that said them stop being sent.
- Newer facts over older ones. "Moved to Denver" replaces "lives in Austin". The model sees what is true now, not both.
- Older messages that matter now. When the user asks about the inspector, Tuesday's messages about the wiring come back word for word, with their dates. Nobody had to mark them important in advance.
Nothing is thrown away. Older messages that are not sent are still stored, and come back when the conversation turns to them.
How Threadless AI does this for you
Before each model call, your agent hands Threadless AI the new message and gets back the request to send: your instructions, the facts from older messages, the older messages that matter to what was just asked, and the latest messages word for word. After the reply, it hands the reply back. Before older messages stop being sent, your own model notes their key facts with the note tool, so the facts are written by the model that was in the conversation.
The result: one conversation can run for days or weeks, token use stays bounded as it grows, and what matters is still in front of the model. How it works shows the request on one example.
Not to be confused with catastrophic forgetting
Searches for "LLM forgetting" mostly find a different problem. Catastrophic forgetting happens in training: a neural network trained on a new task loses what it learned on earlier ones, long described as "an inevitable feature of connectionist models" (Kirkpatrick et al.). It concerns a model's weights.
This page is about an agent's memory at run time: which parts of a conversation the model sees on each call. The model's weights do not change; nothing is trained. Forgetting here is a choice about context, made on purpose.
Questions people ask
Why not just send the whole conversation?
It costs more on every turn, it slows every answer, models use the middle of a long prompt poorly, and eventually it does not fit in the context window.
Does an agent lose information when it forgets?
It depends on the design. In Threadless AI, no: older messages stop being sent word for word, but their facts stay in view, and everything stays stored and can come back. Systems that delete old data, or replace it with a summary, do lose it.
What should an agent keep in view?
The latest messages word for word, the facts from older messages, and the older messages that matter to what was just asked. Where facts conflict, the newer one.
Does forgetting improve agent performance?
There is evidence that it can. In one study, agents that stored only selected experiences beat agents that stored everything on all four tasks, some by more than 20 points, with smaller memories.
What is catastrophic forgetting?
A training problem: a neural network that learns a new task loses performance on old ones as its weights change. It is unrelated to how an agent manages its conversation memory at run time.