Large Language Model Based Long Context Modeling Papers and Blogs

This repository curates papers and blogs on long-context language modeling, covering surveys; efficient attention; KV-cache optimization; recurrent transformers and state-space models; position encoding & length extrapolation; long-context training; long-term memory; retrieval-augmented generation; in-context learning; context and model compression; long reasoning (long CoT); long video & image; long-horizon agents; long-text generation; inference acceleration; benchmarks & evaluation; and technical reports.

🔥 Must-read papers for LLM-based Long Context Modeling.

🔥⚡🔥 Thanks for all the great contributors on GitHub!

🚀🤝🚀 I have the privilege of joining [LCLM-Horizon] and collaborating with them on providing a very complete and comprehensive scholarly survey (A Comprehensive Survey on Long Context Language Modeling) and repository (A-Comprehensive-Survey-For-Long-Context-Language-Modeling) dedicated to Long Context Language Modeling. I look forward to collaborating with them to advance research and deepen understanding in this area!

If you find our repository and survey useful for your research, please consider citing the following paper:

@article{liu2025comprehensive,
  title={A Comprehensive Survey on Long Context Language Modeling},
  author={Liu, Jiaheng and Zhu, Dawei and Bai, Zhiqi and He, Yancheng and Liao, Huanxuan and Que, Haoran and Wang, Zekun and Zhang, Chenchen and Zhang, Ge and Zhang, Jiebin and others},
  journal={arXiv preprint arXiv:2503.17407},
  year={2025}
}

Contents

• 📢 News
  • Week Papers
  • Month Papers
• 📜 Papers
  • 1. Survey Papers
  • 2. Efficient Attention
  • 3. KV-Cache Optimization
  • 4. Recurrent Transformers
  • 5. State Space Models & Hybrids
  • 6. Position Encoding & Length Extrapolation
  • 7. Long-Context Training
  • 8. Long-Term Memory
  • 9. Retrieval-Augmented Generation
  • 10. In-Context Learning (Many-shot / Long-ICL)
  • 11. Context Compression
  • 12. Model Compression for Long Context
  • 13. Long Reasoning (Long CoT)
  • 14. Long Video & Image
  • 15. Long-Horizon Agents
  • 16. Long-form Text Generation
  • 17. Inference Acceleration & Serving
  • 18. Benchmarks & Evaluation
  • 19. Technical Reports (Long-Context Models)
  • 20. Blogs & Tutorials
• Acknowledgements
  • Contributors
  • Star History

📢 News

Week Papers

• [2026.08.14]

  • Paper: SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning
  • Paper: KV Cache Compression Through the Lens of Transform Coding
  • Paper: MemoryLake on MemoryArena: A Matched Study of Agent Memory Backends
  • Paper: AgentRewind: Recoverable Execution for Long-Horizon LLM Agents
  • Paper: ScienceFlow: A long-horizon agent for ML research, scientific discovery and beyond
  • Paper: MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning
  • Paper: Handover of In-Context Learning State Across Session Boundaries

• [2026.08.13]

  • Paper: The Query Knows What to Forget: A Second Erase Direction for Linear Attention
  • Paper: When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation
  • Paper: RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory
  • Paper: vToken: Token-Level Virtualization for Reclaimable KV Caches
  • Paper: SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention
  • Paper: LycheeMemory V2: Efficient Long-Term Memory for LLM Agents via Semantic Segment-Level Consolidation
  • Paper: AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design
  • Paper: PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives
  • Paper: Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development
  • Paper: Beyond Retrieval: Query-Conditioned Reuse of Long-Horizon Agent Trajectories
  • Paper: CoverPrune: Coverage-Driven Token Pruning for 3D VLMs via Optimal Transport
  • Paper: NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video
  • Paper: EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory

• [2026.08.12]

  • Paper: EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory
  • Paper: Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
  • Paper: MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
  • Paper: Disentangling the Expressivity of RoPE
  • Paper: LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration
  • Paper: Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
  • Paper: The Sleeping Agent: What Gist-Based Context Compression Loses and Why
  • Paper: Governed Persistent Memory: Source-Bound State Semantics and Fail-Closed Release for Long-Horizon Agents
  • Paper: Towards a Formal Definition of Agent Memory: Basis, Span, Optimality, and the Sequential Memory Problem
  • Paper: Beyond Memory: A Transactional Continuity Kernel for Long-Lived AI Agents
  • Paper: LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation
  • Paper: Hybrid Gated Attention
  • Paper: Claim-Level Reliability Assessment for Efficient Test-Time Reasoning
  • Paper: Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling

• [2026.08.11]

  • Paper: Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks
  • Paper: Neural Introspection Gating for Adaptive KV-Cache Reuse in Vision-Language-Action Models
  • Paper: ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover
  • Paper: When Vision Becomes Text: Visual Token Pruning via Cross-Modal Residual Guidance in VLMs
  • Paper: SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesis
  • Paper: Self-Correcting Long-Horizon Search Agents via Tree-Structured Memory
  • Paper: StreamFlow: Dynamic Memory Flows for Streaming Video Understanding
  • Paper: InSight-doc: Agentic Visual Perception for Long-Document Understanding
  • Paper: R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video
  • Paper: ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

• [2026.08.10]

  • Paper: Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
  • Paper: MixFormer: Linear Transformer with Mixture of Memory Experts
  • Paper: KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models
  • Paper: MESA:Task-Adaptive Multi-Structure Evidence Selection for Long-Horizon Agent Memory
  • Paper: Not All Visual Tokens Are Equally Safe to Remove:Consequence-Sensitive Visual Token Compression
  • Paper: Evo-Bench: Can Language Models Improve Agent Harness?
  • Paper: BDH-CQ: In-Context Learning with Recurrent Latent Reasoning
  • Paper: Motif 3: Technical Report

• [2026.08.09]

  • Paper: DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
  • Paper: RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation
  • Paper: VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling
  • Paper: Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling
  • Paper: VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference
  • Paper: Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses
  • Paper: Not Worth Another Token: Marginal Value Estimation for Efficient Deep Research Agents

• [2026.08.08]

  • Paper: OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
  • Paper: SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Coding
  • Paper: CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents
  • Paper: SuperLocalMemory 4.0: The Governed Memory Operating System for AI Agents

• [2026.08.07]

  • Paper: CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG
  • Paper: HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
  • Paper: Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compression
  • Paper: Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
  • Paper: StateFlow: Sequence Pipeline Parallelism for Long-Context Modeling with Linear Recurrence
  • Paper: The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents
  • Paper: RoRA: Role-Oriented Regional Allocation for Visual Token Pruning in MLLMs
  • Paper: Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
  • Paper: Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding
  • Paper: An AI4AI Framework for Visual Token Pruning
  • Paper: DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understanding
  • Paper: MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents
  • Paper: Long-Horizon Agent Trajectory Attribution: A Unified Benchmark and Fine-Grained Annotation Framework
  • Paper: MemPrism: Task-Conditioned Relational Memory Views for Long-Horizon Agents
  • Paper: HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses
  • Paper: Explicit, Not Longer: What Makes Epistemic Stance Survive Memory Compression
  • Paper: CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing
  • Blog: Efficient Decode Context Parallelism with vLLM for Long Context Workloads

• [2026.08.06]

  • Paper: Retrofitting Linear Attention into Diffusion Language Models
  • Paper: Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding
  • Paper: Toward Reliable Context Compression for Long-Horizon Agents: An Empirical Study of Execution Instability
  • Paper: StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding
  • Paper: One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding
  • Paper: TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
  • Paper: Runtime Observability for Heterogeneous Attention Memory
  • Paper: Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning

• [2026.08.05]

  • Paper: Recursive Synthesis for Long-Horizon Terminal Tasks
  • Paper: QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
  • Paper: OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
  • Paper: Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression
  • Paper: Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning
  • Paper: MemoryCPT: An End-to-End Agent Memory Framework for Cost-Performance Trade-off
  • Paper: Caching for the Future: Scrub Jay Episodic Memory Principles for Agent Memory Systems
  • Paper: Not All Redundant Tokens Are Alike: Analyzing Visual Token Pruning through Token Roles
  • Paper: ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment
  • Paper: EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
  • Paper: Thinking with Anchors: Grounded and Efficient Document Reasoning
  • Paper: Chained Recursive Language Models for Multi-Iteration Reasoning
  • Paper: Training-Free Hashing-Based Attention via Binary Principal Components

• [2026.08.04]

  • Paper: Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
  • Paper: TimeRLM: Recursive Language Models Enable Precise Anomaly Localization in Long-Context Time-Series
  • Paper: Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks
  • Paper: TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
  • Paper: PI-Mem: Pushing Long-Context Reasoning to 3.6M Tokens with Parallel-Iterative Memory
  • Paper: Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse
  • Paper: Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention
  • Paper: SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
  • Paper: RUTA: Principled Visual Token Allocation via Rate-Utility Optimization
  • Paper: Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models
  • Paper: GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Models
  • Paper: StreamDAM: Presence-Aware Memory for Real-Time Streaming Video Object Segmentation
  • Paper: When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding
  • Paper: Muon Meets Mamba: Spectral Optimization for State Space Models
  • Paper: Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
  • Paper: EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners
  • Paper: When Do Fewer Visual Tokens Accelerate Multimodal Inference? A Break-Even Study Across Decision Locations and Hardware
  • Paper: Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin
  • Paper: Interpretable Adaptive Sampling for LLM Test-Time Scaling
  • Paper: SAKI: Score-Aware Low-Rank Key Indexing with Random-Matrix Noise Correction for KV Retrieval
  • Paper: OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents

Month Papers

📜 Papers

Paper entries live under papers/ so this README stays under GitHub’s homepage size limit. For an interactive chapter reader (search + in-page paper cards), open the project homepage.

Acknowledgements

Please contact me if I miss your names in the list, I will add you back ASAP!

Contributors

Star History