# Infinite context for LLMs: what it is, how to get it

> No model has an infinite context window. What infinite context means in research (Infini-attention, EM-LLM) and how agents get it in practice today.

**Is there an LLM with infinite context?** No. Every model you can call today has a finite context window, and every long conversation reaches the end of it. "Infinite context" means two different things depending on who says it:

- **In research**, it is an architecture that lets a model process inputs of unbounded length with bounded memory: Infini-attention, StreamingLLM, EM-LLM, state space models.
- **In practice**, it is what a conversation is like when it never hits the wall: the agent keeps what matters in view, keeps the facts of the rest, and recalls details when they come up.

Our position: **the window is finite; the conversation isn't.** Infinite context is not a bigger window. It is assembled, call by call, from a finite one. This page covers both meanings. For memory in general (types, lifecycle, evaluation, tools), see [AI agent memory](https://threadlessai.com/guides/ai-agent-memory).

## How big are context windows today?

Large, and finite. On 12 November 2026:

- **OpenAI** lists a 1.05M-token context window for each of its current flagship models ([models](https://developers.openai.com/api/docs/models)). GPT-5 lists 400,000, with 272,000 of input ([GPT-5](https://developers.openai.com/api/docs/models/gpt-5)).
- **Google** says "many Gemini models come with large context windows of 1 million or more tokens" ([long context](https://aistudio.google.com/docs/long-context)).

A million tokens is a lot of text, and a conversation that runs for months, with tool output, still passes it. The question is not whether a window is big enough today but what happens on the turn it isn't.

## What does infinite context mean in research?

Several lines of work aim at inputs longer than any fixed window. None of them is a switch you can turn on in a hosted API.

| Approach | Idea | What it gives up |
|---|---|---|
| **Infini-attention** (Google, 2024) | Adds a compressive memory to attention, aiming at "infinitely long inputs with bounded memory and computation"; tested on 1M-token passkey retrieval and 500K-token book summarisation ([arXiv](https://arxiv.org/abs/2404.07143)) | Older context survives only in compressed form |
| **StreamingLLM** (2023) | Keeps the first tokens as "attention sinks" plus a sliding window, so models "generalize to infinite sequence lengths without any fine-tuning"; stable on 4M+ tokens ([arXiv](https://arxiv.org/abs/2309.17453)) | The attention window is still finite; text that leaves it is not recalled |
| **EM-LLM** (ICLR 2025) | Segments tokens into episodic events by surprise and retrieves them by similarity and time; retrieval across 10 million tokens ([arXiv](https://arxiv.org/abs/2407.09450)) | Needs access to the model's attention; not available through an API |
| **State space models** (Mamba, 2023) | Linear-time sequence models, with performance that "improves on real data up to million-length sequences" ([arXiv](https://arxiv.org/abs/2312.00752)) | A fixed-size state summarises the past |
| **Virtual context** (MemGPT, 2023) | Treats the window like an operating system treats RAM, moving data between fast and slow memory ([arXiv](https://arxiv.org/abs/2310.08560)) | The model must manage its own memory with tool calls |

Results do not always carry over. Hugging Face tried to reproduce Infini-attention and concluded that its "performance gets worse as we increase the number of times we compress the memory", recommending other methods for extending a pretrained model's context ([Hugging Face](https://huggingface.co/blog/infini-attention)).

The first four change the model. If you call OpenAI's API, you cannot use them; you use the window you are given. The fifth works at the system level, around the model, and that is the line Threadless AI works in.

## Why is a longer window not the same thing?

Even when the window is long enough, sending everything has costs that grow with the conversation:

- **Cost per turn.** Input tokens are billed on every call. A conversation of a million tokens pays for a million tokens each turn.
- **Latency.** More input means a longer wait before the first token.
- **Attention.** Models use long inputs unevenly. Performance "is often highest when relevant information occurs at the beginning or end of the input context" ([Lost in the Middle](https://arxiv.org/abs/2307.03172)), and in Chroma's test of 18 models, "performance grows increasingly unreliable as input length grows" ([context rot](https://www.trychroma.com/research/context-rot)).
- **No sense of what is current.** A window holds last week and last minute side by side, with nothing to say which one matters now.

## Infinite context as a system

Infinite context for an agent is the experience of the conversation: the user talks to the agent for months, the agent keeps track of what matters, and old things come back when they come up. Threadless AI assembles it and hands it to you as one service. Before each model call you send what is new; you get back the request, history and all, ready to submit to OpenAI.

No single feature gives a conversation infinite context. Six together do:

| Part | What it does | Without it |
|---|---|---|
| **Bounded size** | Every request is sized for a 200k-token context window, with room for the reply; token use stays bounded as the conversation grows | The call fails, or the oldest turns are cut blindly |
| **The latest messages, word for word** | The newest part of the conversation is sent as it happened | The model loses the thread of what was just said |
| **Facts kept** | Before older messages stop being sent word for word, your model notes their key facts with a note tool; they stay in view | The agent forgets what it learned along with the turns that taught it |
| **Newer facts win** | A newer fact replaces the one it contradicts: "moved to Denver" replaces "lives in Austin" | The agent acts on what used to be true |
| **Recall** | When a message touches something older, the older messages and facts about it come back, with their dates, before the message | Old details are gone for good, or buried in every prompt |
| **A local fallback** | If we cannot be reached, the SDK builds the request from recent turns and the call goes ahead | Memory becomes a way for the agent to break |

The user does not see any of these. They see an agent that remembers a delivery date three weeks later, does not repeat a question they answered, and never says the conversation is too long.

## Why it has to be assembled

Each familiar approach to long conversations does one of the six:

- **Truncation** keeps requests under the window, and nothing else.
- **Summaries** keep the gist, and drift.
- **Retrieval** brings old messages back, without any sense of what is current.
- **Bigger windows** delay the problem, cost more per call, and attend less well to what matters now.

Infinite context needs all of them working together, on every call, with one decision about what the model sees. That decision is the hard part, and it is the part Threadless AI owns. [Context window management](https://threadlessai.com/guides/context-window-management) compares the approaches side by side.

## Approaches compared

| | Where it lives | Available today | Cost per call | What it loses |
|---|---|---|---|---|
| Bigger window | The model | Yes, up to about 1M tokens | Grows with the conversation | Attention to the middle; still has an end |
| Architectural "infinite" attention | The model | Research | Bounded, if you train or host the model | Detail of compressed or evicted context |
| Compaction | OpenAI's server ([compaction](https://developers.openai.com/api/docs/guides/compaction)) | Yes, Responses API | Bounded | Readability; details do not come back on demand |
| Memory store plus your own prompt code | Your code and a vendor | Yes | Bounded if you cap it | Whatever your prompt code does not handle |
| Assembled memory (Threadless AI) | A service; you submit the request | Yes, OpenAI formats | Bounded | Older messages, word for word, from the request; they come back when the conversation turns to them |

## What you change in your code

Three calls around the model call you already make:

```ts
const request = await memory.chat.decorate(memoryId, newMessages, { model, tools });
const reply = await openai.chat.completions.create(request);
memory.chat.report(memoryId, reply);
```

1. **decorate** before the call: send only what is new since the last call.
2. **Submit** the request to OpenAI with your own key.
3. **report** the reply. It returns at once.

Add the note tool to your tools and answer its calls with `noted`. That is the whole integration. The [quickstart](https://threadlessai.com/docs/quickstart) has the loop in TypeScript and Python, for Chat Completions and Responses.

## How this differs from a memory store

Most memory services are stores: you add messages, search them, and decide where the results go in your prompt. You still make the hard decision, what fits the window, on every call.

| | A memory store | Threadless AI |
|---|---|---|
| What you get back | Search results to place in a prompt | The request, ready to submit |
| Who decides what fits the window | You | We do, on every call |
| Who writes the facts | Usually the store's own model | Your model, from the turns it was in |
| A sense of what is current | Usually none, or a ranking tweak | Newer facts replace older ones; recalled messages carry their dates |
| When the service is down | Your call fails or goes without memory | The SDK builds the request from a local tail |

## When it is not what you need

- **One-shot calls** with no conversation have nothing to remember.
- **Facts shared across a user's conversations.** A Threadless AI memory is one conversation, and recall never crosses memories. See [Memory per conversation, not per user](https://threadlessai.com/guides/memory-per-conversation).
- **Models outside OpenAI's formats.** The request comes back as Chat Completions or Responses.

## Questions people ask

### Is there an LLM with infinite context?

No. Every model you can call has a finite window; the largest offered by OpenAI and Google are around a million tokens. Research architectures such as Infini-attention aim at unbounded input, but they are not something you can switch on in a hosted API.

### What is Infini-attention?

A 2024 Google technique that adds a compressive memory to a Transformer's attention, so it can process very long inputs with bounded memory. The paper reports results on 1M-token passkey retrieval and 500K-token book summarisation; a Hugging Face reproduction found it degraded as memory was compressed more often.

### Does Gemini have infinite context?

No. Google describes many Gemini models as having windows of 1 million or more tokens. That is large, but finite.

### Will infinite context windows replace RAG?

Not on cost or reliability. Sending a whole corpus on every call pays for all of it each time, and models use very long inputs unevenly. Retrieving the passages a question needs remains cheaper, and usually better.

### How do agents get infinite context in practice?

By managing a finite window: keeping recent and relevant history in view, writing down facts from what leaves, and recalling details when they come back up. That can be built from truncation, summaries and retrieval, or used as a service.

### Is infinite context the same as long-term memory?

Closely related. Long-term memory is what the agent keeps across a conversation that outgrows the window; infinite context is the experience that memory gives the user. See [AI agent memory](https://threadlessai.com/guides/ai-agent-memory).

## Related

- [AI agent memory: long-term memory explained](https://threadlessai.com/guides/ai-agent-memory)
- [How to stop resending the whole chat history to OpenAI](https://threadlessai.com/guides/stop-resending-chat-history)
- [Why agents should forget](https://threadlessai.com/guides/why-agents-should-forget)
- [How it works](https://threadlessai.com/how-it-works)