ContextBench

A diagnostic benchmark of four synthetic tasks that tests Context Management on its own, separate from reasoning and knowledge. It was introduced in the Context Language Models paper (Shao et al., 2026).

The input arrives as a stream of operations, one user message each, so everything lands in the agent's context before it can act on it. Over an episode the stream adds up to as much as 24 times the 32K context limit, which the authors call context pressure. The tasks need no search or inference, so an agent that could keep everything would answer every query.

• Needle Retention asks the agent to keep specific lines word for word while deleting filler.
• Sudoku Sketchpad asks it to keep a 16×16 Sudoku board up to date as moves stream in, one per turn.
• KV Store sends batches of 100 key-value pairs, then asks for exact values later.
• Log Triage streams service logs, then asks lookup and count questions about them.

Figure 2 from (Shao et al., 2026), CC BY 4.0. All methods use GPT-5.4 with a 32K context limit.

With GPT-5.4, none of the existing methods (Codex-style summary, Context Folding, RLM, Self-Compact and ACM) were perfect, even on these simple tasks. Summaries lose or hallucinate needles and Sudoku cells, and coding tools can offload values to disk but can't evict them from the live context. CLM stays near 100% at every level of pressure.

Every method gets a task-specific skill explaining how to use its own tools on each task, so ContextBench tests whether a method can carry out a context management strategy, not whether the model can come up with one.