Add memory to a chatbot (OpenAI, streaming included)

A chatbot built on the OpenAI API remembers nothing by itself. Each request carries the history the model sees, so "memory" starts as a list your code keeps and resends. That works until the list outgrows the context window, gets expensive, or has to survive a restart.

This page adds memory to a chatbot in three steps without streaming, then covers what streaming changes: when to save the reply, how to get token usage out of a stream, and what to do when the user presses stop.

The chatbot without memory

The usual starting point keeps the history in a list and sends all of it every time:

history = [{"role": "system", "content": "You are a helpful assistant."}]

def chat(text: str) -> str:
    history.append({"role": "user", "content": text})
    reply = openai.chat.completions.create(model="gpt-5", messages=history)
    history.append({"role": "assistant", "content": reply.choices[0].message.content})
    return reply.choices[0].message.content

Three things go wrong as it grows: the list lives in one process and one user's session, every turn resends and pays for every earlier turn, and eventually the request no longer fits. How to stop resending the whole chat history covers trimming and summarising. The rest of this page replaces the list.

Add memory to a chatbot in three steps

1. Create a memory per conversation

A Threadless AI memory holds one conversation. Create it when the conversation starts and keep its id next to your conversation id.

from threadlessai import Threadless

memory = Threadless()  # reads THREADLESS_API_KEY
memory_id = memory.memories.create(metadata={"conversation": "c_123"})["id"]

2. Send only what is new, submit what comes back

decorate takes what is new since the last call and returns the full request. You submit it with your own OpenAI key, then report the reply.

from openai import OpenAI
from threadlessai import NOTE_RESULT, Threadless, is_note_call

openai = OpenAI()  # reads OPENAI_API_KEY
memory = Threadless()
tools = [memory.chat.note_tool]  # plus your own tools

def send(memory_id: str, text: str):
    new_messages = [{"role": "user", "content": text}]
    while True:
        request = memory.chat.decorate(memory_id, new_messages, {"model": "gpt-5", "tools": tools})
        reply = openai.chat.completions.create(**request)
        memory.chat.report(memory_id, reply)

        message = reply.choices[0].message
        if not message.tool_calls:
            return message.content
        new_messages = [message.model_dump(exclude_none=True)]
        for call in message.tool_calls:
            output = NOTE_RESULT if is_note_call(call.function.name) else run_tool(call)
            new_messages.append({"role": "tool", "tool_call_id": call.id, "content": output})

Send your system message on the first call and again only when it changes; it is kept and always leads the request.

3. Answer the note tool

When part of the history is about to leave the model's view, the request asks the model to write that history's facts with one call to the note tool. Nothing runs: answer every note call with noted, as the loop above does, and call decorate again. The three calls explains each part, and the quickstart has the same loop in TypeScript and for the Responses API.

That is the whole chatbot. The request stops growing with the conversation, the facts of old turns stay in view, and details come back when the user mentions them.

One memory per chat, many users

Key memory by conversation, not by process or by user:

• Store the memory id beside your conversation id, in the same row of your database. You can also put your ids in the memory's metadata (up to 16 string pairs).
• A user with five chats has five memories. A memory's id is its only scope, and recall never crosses memories, so one chat's facts never surface in another.
• Durable facts about the user (name, plan, preferences) belong in your instructions, which you control.

See Memory per conversation, not per user for why.

What changes when you stream?

Streaming does not change what you send before the model call. decorate returns the request; you add stream: true and send it. What changes is the reply: it arrives in pieces, and the memory needs the whole of it, with the token usage, to store it and keep the request size on target.

The rule: report the final response once the stream ends.

Streaming with the Responses API

The response.completed event carries the full response; OpenAI's SDK types define it as the event "emitted when the model response is complete", holding the response (openai-python). Report that.

const request = await memory.responses.decorate(memoryId, newItems, { model: 'gpt-5', tools });
const stream = await openai.responses.create({
  ...request,
  stream: true,
} as OpenAI.Responses.ResponseCreateParamsStreaming);

for await (const event of stream) {
  if (event.type === 'response.output_text.delta') write(event.delta);
  if (event.type === 'response.completed') memory.responses.report(memoryId, event.response);
}

request = memory.responses.decorate(memory_id, new_items, {"model": "gpt-5", "tools": tools})
stream = openai.responses.create(**request, stream=True)

for event in stream:
    if event.type == "response.output_text.delta":
        write(event.delta)
    elif event.type == "response.completed":
        memory.responses.report(memory_id, event.response)

Streaming with Chat Completions

Chat Completions streams chunks, and includes usage only when you ask for it. Request stream_options: { include_usage: true }, assemble the chunks into one completion, and report that. The OpenAI SDKs can assemble it for you: in Node, openai.chat.completions.stream(...) returns a stream whose finalChatCompletion() "resolves with the final chat completion that was received from the API" (openai-node helpers); in Python, get_final_completion() "returns the accumulated" completion (openai-python helpers).

const request = await memory.chat.decorate(memoryId, newMessages, { model: 'gpt-5', tools });
const stream = openai.chat.completions.stream({
  ...request,
  stream_options: { include_usage: true },
} as unknown as OpenAI.ChatCompletionCreateParamsStreaming);

for await (const chunk of stream) write(chunk.choices[0]?.delta?.content ?? '');
memory.chat.report(memoryId, await stream.finalChatCompletion());

request = memory.chat.decorate(memory_id, new_messages, {"model": "gpt-5", "tools": tools})

with openai.chat.completions.stream(**request, stream_options={"include_usage": True}) as stream:
    for event in stream:
        if event.type == "content.delta":
            write(event.delta)

memory.chat.report(memory_id, stream.get_final_completion())

If you assemble chunks yourself instead, build one completion object with the full message, any tool calls, and the usage from the last chunk, then report it.

How do you get token usage from a stream?

• Chat Completions. With include_usage, "an additional chunk will be streamed before the data: [DONE] message"; its usage field "shows the token usage statistics for the entire request", and "the choices field will always be an empty array". Every other chunk has usage: null (Chat Completions reference). Code that reads chunk.choices[0] must allow for that last, empty chunk.
• Responses. The response in response.completed is the full response object, whose usage holds input, output and total tokens (openai-python).

Streaming does not change what the call costs: the same tokens are billed whether they arrive at once or in pieces.

Why the usage matters

We estimate the size of every request we build, and correct the estimate with the token count your provider reported on the previous call. A report without usage still stores the reply, but the correction for your model waits for the next report that has it.

Tool calls while streaming

Tool calls stream too. Once the stream ends, handle the calls in the final response as you would without streaming: answer note calls with noted, run your own tools, and call decorate again with the reply and the results.

If the user stops the stream

A user who presses stop leaves you with part of a reply and no final response. OpenAI's reference warns that "if the stream is interrupted, you may not receive the final usage chunk" (Chat Completions reference). In Node, an aborted stream makes finalChatCompletion() reject with APIUserAbortError (openai-node helpers); for Responses, finalResponse() can resolve with a partial response "whose status is not completed" (openai-node streaming).

report takes the complete response, so don't report a stopped stream. Decide instead what the conversation should record:

• Keep what the user saw. On the next turn, send the partial text as an assistant message before the user's new message. decorate accepts the model's last reply with a new user message; a reply that was never reported is stored as sent.
• Or keep nothing. Send only the user's next message, as if the stopped reply never happened.
• Drop half-streamed tool calls. A tool call without its result makes the next request invalid. Keep the text only.

let shown = '';
let stopped: { role: 'assistant'; content: string } | null = null;
const controller = new AbortController(); // call controller.abort() when the user presses stop

try {
  const stream = openai.chat.completions.stream(params, { signal: controller.signal });
  for await (const chunk of stream) {
    const text = chunk.choices[0]?.delta?.content ?? '';
    shown += text;
    write(text);
  }
  memory.chat.report(memoryId, await stream.finalChatCompletion());
} catch (error) {
  if (!(error instanceof OpenAI.APIUserAbortError)) throw error;
  stopped = shown ? { role: 'assistant', content: shown } : null;
}

// Next turn: what the user saw, then what they said.
const newMessages = [...(stopped ? [stopped] : []), { role: 'user', content: nextText }];

If the stopped call was one that asked for notes, nothing is lost: when a run ends without the note it was asked for, that history stays leaving and the next call asks again (The three calls).

What the user sees

Nothing different. report returns at once and the work it starts runs in the background, so the end of the stream is not delayed. If a report is still in flight when the next decorate on the same memory starts, the SDK waits briefly for it, so the next request includes the reply.

Questions people ask

How do I add memory to an OpenAI chatbot?

Keep the conversation somewhere other than one process's list, and decide on every call what the model should see. With Threadless AI, create a memory per conversation, call decorate with what is new before each model call, submit the request it returns, and report the reply.

How does ChatGPT remember things?

ChatGPT is OpenAI's app, and its memory belongs to the app. The API does not share it: each API request carries the history the model sees. See Does the OpenAI API have memory?

Do I need a vector database to give a chatbot memory?

Not to start. A vector database helps you search old messages, but you still have to decide what fits the window and keep facts current. A memory service can do that for you; see Agent memory vs RAG.

How do I keep memory separate for each user?

Create one memory per conversation and store its id with your conversation record. A user's chats each get their own memory, and nothing is recalled across them.

Does streaming change token cost?

No. The same input and output tokens are billed whether the reply is streamed or not. Streaming only changes when the reply arrives, and you need include_usage to see the counts in a Chat Completions stream.

When should I save the assistant's reply when streaming?

After the stream ends, from the final response: the response.completed event for Responses, or the assembled completion for Chat Completions. Saving chunk by chunk stores fragments and misses the usage.

How do I stop the chat history growing forever?

Trim or summarise it before each call, or send only what is new to a memory service that returns a request of bounded size. See How to stop resending the whole chat history.

Related

• Add long-term memory to a Responses API agent
• Memory per conversation, not per user
• The three calls
• Models and formats: streaming