Agent memory vs RAG: differences and when to use each

Agent memory, RAG and vector databases get used as if they were one thing, because they share a technique: embeddings and similarity search. They solve different problems.

• RAG answers "what do the documents say?" The documents exist before the conversation and do not change because of it.
• Agent memory answers "what has this conversation established?" The material is the conversation itself, and it changes on every turn.
• A vector database is a component that either can use. It is not, by itself, either one.

Definitions

• A vector database stores embeddings and finds the nearest ones to a query.
• RAG (retrieval-augmented generation) answers a question from a body of documents: search the documents, put the best passages in the prompt, ask the model. The name comes from Lewis et al. (2020), who described models that "combine pre-trained parametric and non-parametric memory for language generation", the non-parametric part being a dense vector index of Wikipedia (arXiv).
• Agent memory carries a conversation forward: what was said, decided and learned, kept available as the conversation outgrows the context window. See AI agent memory for the full picture.

Side by side

Where does RAG break down as memory?

The common shortcut is to embed every message and, on each turn, retrieve the most similar ones. It fails in predictable ways.

Three conversations where it shows:

• A fact that changed. In week one the user says the launch is on 3 March. In week three: "We moved it to the 17th." A month later they ask "how long until launch?" Both messages are similar to the question. Nothing in a similarity score says which is current.
• "What did we decide last week?" The question has almost no words in common with the decision it is asking about. Retrieval by similarity to the question finds other questions.
• A correction. "Actually, the customer is Acme Ltd, not Acme Inc." Without a step that retires the old name, both stay retrievable, and the wrong one may rank higher because it was mentioned more often.

Memory has to answer different questions from retrieval: what should the model still see now, what was learned from what it no longer sees, and what from the past is relevant to this message.

What Threadless AI does instead

Threadless AI answers those questions for you on every call:

• What stays in view: the latest messages, word for word, and the facts from older messages.
• What is learned is written as facts by your own model, with a note tool, before older messages stop being sent word for word.
• What changed: a newer fact replaces one it contradicts, so "we moved it to the 17th" replaces "the launch is on 3 March".
• What matters now: when a new message turns to something older, the older messages and facts about it come back, with their dates, just before the message. A message that asks about two things gets older messages for both.

In code, the difference is who assembles the prompt. With RAG-as-memory you search and place results yourself:

// Retrieval as memory: you decide what goes where, on every call.
const hits = await vectors.search(embed(text), { topK: 8 });
const messages = [system, { role: 'system', content: format(hits) }, ...lastTurns, { role: 'user', content: text }];
const reply = await openai.chat.completions.create({ model, messages });
await vectors.upsert(embed(text), text);

With Threadless AI you send what is new and submit what comes back:

const request = await memory.chat.decorate(memoryId, [{ role: 'user', content: text }], { model, tools });
const reply = await openai.chat.completions.create(request);
memory.chat.report(memoryId, reply);

Signs your RAG-only agent needs memory

• Users repeat things they told the agent earlier in the same conversation.
• The agent cites an old value after the user corrected it.
• Questions about "earlier", "last time" or "what we agreed" get wrong or vague answers.
• The prompt is full of retrieved chat snippets that are similar to the question but beside the point.
• Long conversations still fail with context-length errors, because retrieval was added but the history was never managed.

Can you use both?

Often, yes, and they should stay separate. An assistant that answers from your product docs and also works with the same user for weeks needs RAG over the docs and memory of the conversation:

• Documents in your retrieval system. Retrieve per question, as you do now.
• The conversation in memory. Threadless AI passes your request fields through unchanged, so retrieved documents you add to your messages travel with the request like any other content.
• Mind the budget. Retrieved passages and history share one window. Cap what retrieval adds.

Is RAG dead?

No. Bigger context windows let you send more documents, but every call pays for everything in it, and models use very long inputs unevenly: performance "is often highest when relevant information occurs at the beginning or end of the input context" (Lost in the Middle). Selecting the passages a question needs is still cheaper and often better. What RAG never was is conversation memory.

Risks: memory drift and poisoned memory

Memory writes; RAG mostly reads. Writing brings risks a document index does not have:

• Drift. Summaries of summaries lose detail and can introduce errors that are then treated as fact.
• Stale facts. A memory that cannot retire a contradicted fact will keep offering both.
• Poisoned input. Anything a user says can become a memory. Treat memories as reference data, not instructions.

Threadless AI keeps facts readable and exportable so you can inspect what was written, has the model that was in the conversation write them, and places recalled facts in a block marked as possible memories, to be used only when relevant (Models and formats).

Questions people ask

Is RAG the same as agent memory?

No. RAG retrieves from documents written before the conversation. Agent memory records and updates what the conversation itself established, and decides what to keep in view as it grows.

Is a vector database memory?

It is storage with similarity search. Memory also needs a way to write what matters, retire what changed, keep a sense of time and decide what fits the window. A vector database can be one part of that.

Does memory replace RAG?

No. If your agent answers from a body of documents, keep RAG for them. Memory covers the conversation; the two sit side by side in one request.

What is the difference between semantic memory and RAG?

Semantic memory is general knowledge an agent holds about the world and itself, often facts drawn from experience. RAG is a technique for fetching passages from a document store at question time. A RAG index can serve as one source of semantic memory, but most of what an agent learns in conversation never enters that index.

Which is cheaper, RAG or memory?

They cost in different places. RAG pays at ingestion and per search. Memory pays to write (extraction or note calls) and to search per turn. On long conversations, managed memory also lowers model costs, because each request stops growing with the conversation.

Can RAG and memory work together?

Yes. Keep documents in your retrieval system and the conversation in memory, and put both in the request with a cap on how much retrieval adds.

Related

• AI agent memory: long-term memory explained
• Context window management for LLM agents
• Does the OpenAI API have memory?
• Concepts