What a Million-Token Context Window Actually Changes

A year ago a 200,000-token context window felt generous. Now it's the floor. Claude, GPT-5.5, and GLM-5.2 all ship a million-token context window as standard, a startup called Subquadratic just raised $29M to push toward 12 million, and the marketing decks have moved on to 10M like it's settled. A million tokens is roughly 40,000 lines of code with documentation, or a full legal case file, or a month of one customer's support history. You can put all of it in a single prompt now. The question every team we work with is asking: does this actually change how we build, or is it a bigger number that mostly changes the bill?

It changes real things. It also doesn't change the things people expect it to. Here's the honest version.

What it actually kills

The clearest win is that a million-token window deletes a pile of plumbing you used to need.

For workloads that reason over one large thing, the chunking logic goes away. The retrieval layer that fetched the "relevant" 4,000 tokens goes away. The multi-session stitching where you summarized turn 8 so turn 12 would fit goes away. If the whole codebase fits, you don't chunk the codebase. If the whole contract fits, you don't embed and rank paragraphs of the contract and pray the right clause made it into the prompt. You hand the model the contract.

That last point is the one people underrate. Retrieval doesn't just add infrastructure. It adds a failure mode. Every RAG pipeline can fetch the wrong context, and when it does the model answers confidently off the wrong material and you get a bug that's miserable to trace. A query that genuinely fits in context skips that entire class of bug. The right answer was in the prompt because everything was in the prompt.

We rebuilt a contract-review feature for a client this spring that used to run a five-stage retrieval pipeline over uploaded agreements. Most agreements are under 60 pages. Under 60 pages fits. We deleted the embedding store, the reranker, and about 600 lines of glue, and the answers got better because the model could see the clause three pages later that referenced the one you asked about. The pipeline had been splitting them apart.

Where it changes agents specifically

The second real change is for agents that run long.

If you've built a multi-step agent, you know the failure that hits around step fifteen: the context fills up, you compress or truncate the history, and the agent forgets why it started. A million-token window pushes that wall much further out. The agent can hold the full task, every tool call, every intermediate result, and keep reasoning without a forced restart or a lossy summary that drops the one detail that mattered.

This is exactly the capability the newest models lead with. The pitch for the frontier coding models in 2026 is sustained autonomy: working for extended stretches, planning across stages, delegating to sub-agents, verifying their own output. None of that works if the agent's memory gets wiped every twenty minutes to fit a small window. The big context is what makes the long-running agent more than a demo.

Where it quietly makes things worse

Now the part the bigger-number marketing skips.

Processing a million tokens is slow. Meaningfully slow. A model chewing through a million tokens of input takes a lot longer than the same model handling 200,000, and the user staring at a spinner doesn't care why. For anything interactive, where a person is waiting on the response, stuffing the context full is often the wrong move. Big context is a batch and async tool first. Overnight document processing, codebase audits, compliance sweeps, report generation. Not the chat box where someone expects an answer in two seconds.

It's also not free. You pay per input token, and a million-token prompt is a million input tokens every single call unless you're caching. On the current frontier models that's real money per request, and it adds up fast at any volume. The team that "solved" their retrieval problem by jamming everything into context sometimes just moved the problem from engineering effort to the monthly invoice.

And big context doesn't mean perfect recall. Models still attend unevenly across a giant window. A fact buried in the middle of 900,000 tokens is not as reliably used as the same fact near the top. "It fits" and "the model will use it well" are different claims. For a lot of real workloads a semantic graph or a light retrieval pass that puts the right material near the front still beats dumping everything in and hoping.

So is RAG dead?

No. This is the take we keep arguing with, so let's be direct.

Long context kills RAG for the workloads where the relevant material is one coherent thing that fits: one codebase, one contract, one case file, one customer's history. For those, retrieval was always a workaround for a window that was too small, and now that the window grew, the workaround can go.

Retrieval is very much alive for everything else. When your knowledge base is 50 gigabytes of documents, it does not fit in a million tokens and it never will. When you need an answer in 300 milliseconds, you can't afford to process a million tokens to get it. When you want to cite the exact source document, retrieval gives you that for free and a giant prompt doesn't. When cost per query matters at scale, pulling the right 5,000 tokens beats paying for a million every call.

The actual 2026 pattern isn't "long context replaced RAG." It's "pick by workload." Small coherent corpus, latency-tolerant, correctness-critical: long context, skip retrieval. Huge corpus, latency-sensitive, cost-sensitive, citation-required: keep retrieval, and use the bigger window to retrieve fewer, fatter chunks so the model has more room to reason over each one. Most real products end up with both, used in different places. Anyone telling you to rip out your whole retrieval stack is selling something.

How to actually use the big window without lighting money on fire

A few rules we're applying on client builds right now.

Cache the stable part. Most AI features have a long fixed prefix, the system prompt, the codebase, the document, and a short variable suffix, the user's actual question. That's the exact shape prompt caching was built for, and the discount on cached input tokens is large. If you're paying full price for a million-token prompt on every call, you're doing it wrong. We covered this in adding AI to an existing SaaS and it matters even more once your prompts are this big.

Put the important thing near the front. Until recall across a full window is perfect, and it isn't, treat position as a tool. The clause you most need the model to weight goes early, not buried at token 700,000.

Measure cost per request before you ship, not after. A feature that's cheap at 50,000 tokens of context can be twenty times the price at a million. Run the math on your real traffic. Sometimes the answer is a smaller context and a retrieval pass, purely because the frontier price doesn't pencil out at your volume.

Default to async for anything over a few hundred thousand tokens. If a person is waiting, don't make them wait on a million-token read. Move it to a background job, show progress, deliver the result when it's done.

The real shift

The interesting thing about the million-token window isn't that the number got big. It's that "how do I fit this in the model" stopped being the hard part of building most AI features. For a few years that constraint shaped every architecture: chunk, embed, retrieve, rerank, stitch, summarize. A lot of that was scaffolding around a window that was too small, and a chunk of it can come down now.

What's left is the part that was always the actual work. What does the feature need to do, for which user, at what cost, fast enough to matter. The window getting bigger didn't answer any of those. It just stopped giving you an excuse to avoid them.

Trying to figure out whether your AI feature should use long context, retrieval, or both, and what it'll cost at your volume? Book a free strategy call. We'll look at the workload and tell you which architecture actually fits, not which one is trendy. See how we build AI systems or our work.