StreamingLLM

StreamingLLM retains the KV states of initial attention-sink tokens and recent tokens while evicting older states. This supports continued streaming inference with a bounded cache; it does not preserve access to the entire earlier conversation.

Source: Efficient Streaming Language Models with Attention Sinks.

Referenced in An Empirical Study of Harness Design for Coding Agents.