AI agent memory: long-term memory for agents explained

AI agent memory is everything an agent keeps beyond what fits in a single model call: what was said, decided and learned, stored outside the model and brought back into the context window when it is needed. Long-term memory is the part that outlives the window, so the agent still knows on turn 500 what the user told it on turn 5.

Language models are stateless. Each API call sees only the tokens in that request, and the model's weights do not change between calls. Whatever an agent "remembers" is something its code, or a service, put into the request. Agent memory is the system that decides what that is.

This page covers the types of memory, how memory is written, recalled and forgotten, why a bigger context window does not replace it, how to evaluate it, and the tools that provide it. We make Threadless AI, one of those tools; where it fits is near the end, with the cases where it does not.

Short-term vs long-term memory

Short-term memory is easy until it overflows. Long-term memory is what takes over when it does.

What are the types of agent memory?

Agent memory borrows its vocabulary from cognitive science. The Cognitive Architectures for Language Agents (CoALA) framework uses four kinds (Sumers et al.):

Most "long-term memory" products work on episodic and semantic memory: they keep what happened and distil facts from it. The agent memory glossary defines these and related terms.

Why doesn't a bigger context window replace memory?

Windows have grown to around a million tokens (OpenAI models). Sending everything still falls short:

• Accuracy falls with length. LongMemEval found commercial chat assistants and long-context models showing a "30% accuracy drop on memorizing information across sustained interactions" (Wu et al.). Models also favour the start and end of long inputs over the middle (Lost in the Middle).
• Cost and latency grow every turn. Each call pays for and reads the whole history again.
• No sense of what is current. A window holds a superseded fact and its correction side by side.
• The conversation keeps going. Any window has an end; a conversation that runs for months reaches it.

A window is working memory. Long-term memory decides what goes into it. Infinite context for LLMs covers the research on unbounded windows and why none of it is available through a hosted API.

How is agent memory written?

Every memory system answers three questions about writing: what gets stored, who decides, and when.

• Store everything raw. Keep every message and search it later. Simple, complete, and the search has to do all the work.
• Extract with a separate model. After each turn, a memory service's own model reads the messages and pulls out facts. The agent's code stays simple; the extractor sees the conversation from outside and can miss what mattered.
• Let the agent write. The model that was in the conversation writes its own memories with a tool. It knows what mattered, at the cost of tool calls in the conversation.

Threadless AI uses the third: when history is about to leave the model's view, the request asks the model to write that history's facts with one call to a note tool, each fact one sentence that reads on its own (The three calls). See Let the model write its own notes.

Writing also means consolidating: the same person under two names should be one entity, a restated fact should merge with the one already held, and a correction should retire the fact it corrects. Systems do this with a knowledge graph, with a model that reconciles new facts against old, or not at all.

How is memory recalled?

Recall brings stored memory back into the window when the conversation needs it. The common pieces:

• Search by meaning (embeddings) finds paraphrases.
• Search by words finds names, ids and numbers that embeddings blur.
• Graph traversal follows a relevant fact to related ones about the same entities.
• A judgment pass has a model check candidates for relevance, so the prompt is not filled with near misses.
• Placement: recalled memory goes somewhere in the request, marked as memory rather than instruction.

The design question is who triggers it. Some systems give the agent a "search memory" tool and let it decide; others recall on every turn without being asked. An agent that has to decide to search its memory often doesn't. Threadless AI recalls for every user message without being asked: the older messages and facts that matter to it come back, with their dates, just before the message (Concepts).

How is memory forgotten?

Forgetting is part of the design, not a failure of it. A memory that keeps everything at equal weight crowds the window with what no longer matters. The options:

• Cut by age. Drop the oldest. Simple, and blind to what still matters.
• Score by importance. Keep what a model rated important. Importance changes, and the score usually doesn't.
• Decay. Lower each memory's availability with time, and raise it with use.
• Delete or fade. Removing a memory loses it. Fading it from view keeps it recallable.

There is evidence selectivity helps: in one study, agents storing only selected experiences beat agents storing everything on all four tasks tested, with smaller memories (Xiong et al.). Why agents should forget covers that evidence, and what sending everything costs, in detail.

What should an agent remember?

• Decisions and their reasons. What was agreed, and why.
• Facts the user stated. Names, numbers, dates, constraints, preferences.
• Where the work stands. Done, blocked, next.
• Outcomes of actions. What a tool run found, not its raw output.

And what it should not:

• Raw tool output once its result is recorded.
• Pleasantries and filler.
• Instructions hidden in user content. Anything a user says can become a memory; treat memories as reference data, not as commands.
• Anything you would not want recalled later. If it should not come back, do not store it.

Who is memory for: the user, the agent or the conversation?

Memory needs a scope, and the choice shapes what can go wrong.

• Per user. Facts follow a person across conversations. Useful for preferences; risky when one chat's details surface in an unrelated one.
• Per agent. One agent's memory spans every conversation it has.
• Per conversation. Each conversation remembers itself, and nothing crosses between them.

Threadless AI is per conversation: a memory's id is its only scope, and recall never crosses memories. Durable facts about a user belong in your instructions. See Memory per conversation, not per user.

How do you evaluate agent memory?

Two public benchmarks come up in nearly every comparison:

• LongMemEval tests five abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention) with 500 questions embedded in chat histories that can be scaled up (Wu et al.).
• LoCoMo is a set of very long conversations, averaging about 300 turns and 9K tokens over up to 35 sessions, with question answering, event summarisation and multimodal dialogue tasks (Maharana et al.).

Read vendor scores with care. They are usually run by the vendor, with different answering models, judges and question subsets, and some have been publicly disputed: Zep, for one, argued that Mem0's paper misconfigured Zep on LoCoMo and reported its own rerun (Zep). Mem0 alternatives lists published scores with the setup behind each. We are running both benchmarks against Threadless AI and will publish scores with the method; until then we publish no numbers.

Your own tests matter more than either benchmark:

• Long-range recall. A detail from early on, asked about much later.
• Updates. A fact that changes; the agent must use the new value.
• Abstention. A question about something never said; the agent should say so.
• Cost and size. Tokens per call as the conversation grows.
• Corruption. A user states something false or tries to plant an instruction; check what memory does with it.

Can memory make an agent worse?

Yes. Memory that is badly written or badly recalled hurts more than no memory:

• Stale facts recalled after they changed.
• Drift from summaries of summaries.
• Noise: recalled snippets that match the words but not the moment, crowding out what matters.
• Leakage across conversations when the scope is too wide.

The fixes are the same as above: write few, good facts; retire what changed; recall only what fits the moment; scope narrowly.

Agent memory tools compared

Most tools differ on three questions: who builds the prompt, what a memory covers, and who writes the memories. Facts below come from each vendor's documentation, linked, on the date above.

Where Threadless AI fits

Threadless AI is long-term memory for one long conversation with an OpenAI agent. Before every model call your agent sends what is new; it gets back the request, ready to submit with your own key; it reports the reply after.

const request = await memory.chat.decorate(memoryId, newMessages, { model, tools });
const reply = await openai.chat.completions.create(request);
memory.chat.report(memoryId, reply);

What comes back holds the latest messages word for word, the facts from older messages, and the older messages that matter to what was just asked, with their dates. Older messages stop being sent word for word once your model has noted their facts; newer facts replace older ones; and token use stays bounded, with every request sized for a 200k-token context window. If the service cannot be reached, the SDK builds the request from a local tail of recent turns and your call goes ahead (Failure behaviour).

It is the wrong choice when:

• You need facts shared across a user's conversations. A memory is one conversation.
• Your model is called outside OpenAI's formats. Requests come back as Chat Completions or Responses.
• You must self-host. Threadless AI is hosted only.
• Your conversations end well inside the window. You may not need memory yet.

Questions people ask

What is memory in an AI agent?

What the agent keeps beyond a single model call: past turns, facts, preferences and outcomes, stored outside the model and put back into the request when needed. The model itself is stateless.

What are the types of memory in AI agents?

Commonly four: working memory (the current context window), episodic memory (what happened), semantic memory (facts) and procedural memory (how to do things, in the model's weights and the agent's code).

What is the difference between episodic and semantic memory?

Episodic memory is a record of events: this conversation, that tool run. Semantic memory is general knowledge distilled from them: the customer's name, the agreed budget. Agents usually keep episodes and extract facts from them.

Why not just use a larger context window?

A larger window delays the problem. Every call still pays for and reads everything in it, accuracy falls as inputs grow, and nothing in the window says which version of a fact is current.

How do I evaluate whether my agent's memory is working?

Test long-range recall, updates to facts, abstention on things never said, and token cost as conversations grow. LongMemEval and LoCoMo are the common public benchmarks; your own conversations are the better test.

When should I skip a persistent memory layer?

When conversations are short and end well inside the window, or each call is independent. Sending the history, or truncating it, is enough there.

Can memory make an agent worse?

Yes: stale facts, drifting summaries, irrelevant recall and memories leaking across conversations all degrade answers. Write few good facts, retire what changed, and scope memory narrowly.

Related

• Infinite context for LLMs
• Agent memory vs RAG
• Why agents should forget
• Context window management for LLM agents
• Agent memory glossary
• How it works