Context Engineering for Long-Running AI Agents: Compaction, Memory & Real Numbers (2026)

Most AI agents don't fail because the model is dumb. They fail because by hour two of a long task, the context window has turned into a junk drawer — half-finished tool outputs, stale plans, three different versions of the same file, and a system prompt the model stopped paying attention to 40,000 tokens ago. The fix isn't a bigger context window. It's context engineering: deciding, on every single turn, which tokens earn their place in front of the model and which get evicted.

I run seven content-aggregation sites that fire scheduled AI agents every day — importing 100 to 200 records per run, scraping APIs, writing to MySQL, verifying live URLs. The agents that survive a multi-step run aren't the ones on the most expensive model. They're the ones whose context I keep small and clean. This is the playbook I've settled on after watching plenty of runs die at the 60% mark, plus the specific techniques that frontier labs converged on in 2025–2026.

Context engineering is not prompt engineering

Anthropic draws the line cleanly: prompt engineering is "writing and organizing LLM instructions," while context engineering is "the set of strategies for curating and maintaining the optimal set of tokens during LLM inference." The difference is that prompts are mostly static — you write them once. Context is iterative. On every turn of an agent loop you make a fresh decision about what to include, and a single agent run can make hundreds of those decisions.

When I first built the CVE-import agent for one of my security sites, I treated the prompt as the whole job. Big detailed system prompt, lots of instructions, and then I let the conversation grow unbounded. It worked for the first dozen records and then quietly degraded — the agent started re-fetching pages it had already processed because the evidence that it had processed them was buried 50,000 tokens back. That wasn't a prompt problem. No amount of rewording the instructions fixes a context that's drowning the model in its own history.

Why bigger windows don't save you: context rot

The instinct is to reach for the 1M-token model and stop worrying. That instinct is wrong, and there's a name for why: context rot. As the number of tokens in the window grows, the model's ability to accurately recall any specific token decreases. It's not a cliff — it's a steady erosion.

The architectural reason is worth understanding because it tells you this won't be "fixed" by a future model. Transformers let every token attend to every other token, which produces n² pairwise relationships for n tokens. Double the context, quadruple the relational load the attention mechanism has to resolve. Context is a finite resource with diminishing marginal returns, full stop. A 1M-token window is a ceiling you should rarely approach, not a target to fill.

In practice I treat anything past roughly 50% of the model's window as a warning zone. By the time a long-context model is genuinely full, recall on the middle of the conversation — the classic "lost in the middle" failure — is bad enough that I'd rather pay to compact than pay to keep streaming a bloated prompt. And the economics agree: with current pricing where you're paying $15–$25 per million output tokens on flagship models (Opus-class) and even $12 on a Gemini 3.1 Pro, naive context accumulation on a long-running agent isn't just slower, it's a line item.

The four techniques that actually move the needle

Here's what the field has converged on. None of these are exotic — they're disciplines you bolt onto your agent loop.

1. Compaction

When you near the window limit, summarize the conversation history and reinitialize the loop with the summary instead of the raw transcript. Anthropic's compaction approach generates a summary once input tokens cross a configurable threshold, creates a compaction block, and drops every message block before it. Subsequent turns continue from the compacted context.

The tuning advice that took me a few runs to internalize: start by maximizing recall, then improve precision. Your first summarizer should over-include — capture too much rather than risk dropping the one fact the agent needs. Once you trust that it never loses critical state, start trimming the superfluous content. The single safest, highest-ROI move here is tool result clearing: once a tool result has been consumed and acted on, its raw payload (the full API response, the entire file dump) is dead weight. Clear it and keep only the conclusion. On my import agents, raw API JSON responses were the biggest token hog by far, and clearing them after the row was written cut per-run context roughly in half with zero loss of capability.

Independent research backs the magnitude here. Anchored iterative summarization and failure-driven approaches like ACON report 26–54% memory reduction while preserving 95%+ task accuracy — which lines up almost exactly with what I saw empirically.

2. Structured note-taking (memory outside the window)

Compaction is lossy. The complement is to have the agent write durable notes to a store outside the context window — a scratchpad file, a task ledger, a database row — and pull them back in only when needed. This gives you persistent memory with minimal in-context overhead. The pattern I use: keep the active goal and the "minimum proof needed for the next action" pinned at the top of the prompt, and push raw logs out to an event log the agent can query on demand instead of carrying around.

3. Just-in-time retrieval

Don't pre-load everything the agent might need. Maintain lightweight identifiers — file paths, stored queries, web links — and load the actual data at runtime through tools. This is progressive disclosure: the agent discovers relevant context by exploring, the same way a human engineer opens files as the investigation leads them there, rather than reading the entire repo into their head first. For my scraping agents this was the difference between holding 200 records in context and holding 200 row-IDs plus a "fetch row" tool.

4. Sub-agent architectures

For genuinely big tasks, spin up specialized sub-agents that each work a focused slice with a clean context window, then return only a condensed summary — Anthropic cites 1,000–2,000 tokens — to the orchestrator. The main agent never sees the sub-agent's messy working context, only its distilled conclusion. This is separation of concerns applied to tokens. The orchestrator stays lean; the heavy lifting happens in windows you throw away.

Tool design is context design

One under-appreciated lever: your tools are part of your context budget, and badly designed tools poison it. The rule I now hold to is the one Anthropic states bluntly — if a human engineer can't definitively say which tool should be used in a given situation, an AI agent can't be expected to do better.

Tools should have minimal overlap in functionality, be self-contained, be resilient to error, and be unambiguous about when they apply. Early on I had a generic query_db tool and a get_record tool that overlapped, and the agent would oscillate between them, burning turns and tokens deciding. Collapsing them into one clearly-scoped tool removed an entire class of wasted context. Every redundant tool description is also tokens sitting in your window on every turn — bloated tool definitions are a context tax you pay continuously.

Calibrating the system prompt

Context engineering doesn't mean abandoning the prompt — it means right-sizing it. The failure modes sit at two extremes: brittle, complex hardcoded logic that breaks on the first edge case, and vague high-level hand-waving that gives the model no concrete signal. Aim for the minimal set of information that fully outlines the behavior you expect. On my agents the system prompt is short, the tools are sharp, and the bulk of the intelligence lives in what gets curated into context each turn — not in an ever-growing wall of instructions.

A practical checklist

• Set a compaction threshold well below the window cap — I trigger around 50%, not 90%.
• Clear tool results once consumed — raw API/file payloads are the first thing to evict.
• Pin the active goal at the top; push raw logs to an external event log.
• Hold identifiers, not data — load records just-in-time through tools.
• Summarize for recall first, precision second.
• Audit your tools — any two that overlap are a context leak.
• Fan out to sub-agents for big subtasks; keep only their 1–2K-token summaries.

The bottom line

The agents that run reliably for me aren't the ones with the biggest models or the longest prompts. They're the ones where I treat context as the scarce, expensive, performance-critical resource it actually is. Context rot is architectural — it isn't going away with the next release — so the teams that win in 2026 are the ones who get disciplined about the token lifecycle now: compact aggressively, store state outside the window, retrieve just in time, and design tools you can reason about. Do that and a mid-tier model on a clean context will outlast a flagship model drowning in its own history. I've watched it happen on every run.

FAQ

Is context engineering just RAG? No. RAG is one retrieval technique that can feed context. Context engineering is the broader discipline of managing the entire token lifecycle — what enters, what stays, what gets compacted, what gets evicted — across every turn of an agent loop.

When should I compact versus use a bigger context window? Almost always compact. A bigger window delays the problem and worsens context rot; compaction addresses the cause. Reach for a larger window only when a single indivisible input genuinely won't fit.

Does compaction lose information? Yes, it's lossy by design — that's why you pair it with structured note-taking to an external store. Tune your summarizer for recall first so the losses are superfluous content, not critical state.

What's the simplest first step? Tool result clearing. Find the largest raw payloads your agent holds after it's done acting on them, and drop them. It's the highest token-saving-per-line-of-code change you can make.