Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformers

Yukun Zhang^*, Xueqing Zhou^*

The Chinese University of Hong Kong Fudan University
EMNLP 2025
^*Equal contribution

Continuous-Time Attention models token interactions as solutions of partial differential equations (PDEs) in continuous time, enabling efficient and stable modeling of long-range dependencies in Transformers.

TL;DR: We introduce Continuous-Time Attention, a PDE-guided attention mechanism that treats self-attention as a continuous-time dynamical system, improving long-sequence modeling efficiency and stability for Transformers.

System Perspective (Inference/Serving)

From a systems viewpoint, CTA's PDE refinement can be interpreted as a local stencil token-mixing operator (e.g., diffusion Laplacian). Each refinement step is linear in sequence length and kernel-friendly, making it a natural candidate for long-context prefill pipelines.

Below we report synthetic microbenchmarks (forward-only) illustrating how latency and peak memory scale with sequence length. The figures are generated by scripts under benchmarks/.

Evidence snapshot (A800-80GB, bf16, B=1, D=512, heads=8, diffusion, steps=4):
L=4096: baseline p50 0.607 ms → transformer+pde p50 0.988 ms (~1.63×)
L=16384: baseline p50 4.108 ms → transformer+pde p50 5.688 ms (~1.38×)
Profiler (L=4096): cta_transformer_layer ~5.953ms, cta_pde_loop ~5.607ms, transpose/layout total ~0.554ms self CUDA.

System Efficiency: Latency vs Sequence Length

Latency vs Sequence Length (prefill, forward-only)

System Efficiency: Peak Memory vs Sequence Length

Peak Memory vs Sequence Length (prefill, forward-only)

Abstract

Transformers achieve state-of-the-art performance on a wide range of sequence modeling tasks, but their quadratic attention complexity and discrete-layer parameterization make long-sequence modeling both expensive and difficult to analyze. We propose Continuous-Time Attention, a PDE-guided formulation of self-attention that treats token interactions as trajectories of a continuous-time dynamical system governed by partial differential equations. This view allows us to derive new attention mechanisms with better control over information propagation, improved stability for long-range dependencies, and favorable computational properties for long sequences. We instantiate our framework in several long-sequence benchmarks, where Continuous-Time Attention attains competitive or superior performance to strong Transformer baselines while offering a principled lens on how information flows across depth and positions.

Results

Figure 6: Character-level IMDb results showing training/validation loss and validation accuracy across epochs. Continuous-Time Attention demonstrates improved training stability and better validation performance compared to baseline models.

Figure 7: PDE-Longformer vs vanilla Longformer

Figure 7: Comparison between PDE-Longformer and vanilla Longformer on WikiText-103 dataset. Left: Training/validation loss over 20 epochs. Right: Perplexity trend (lower is better). PDE-Longformer shows superior performance with more stable training dynamics.

Figure 11: Analysis of sequence length impact on model performance. Continuous-Time Attention maintains consistent performance improvements across different sequence lengths, demonstrating its effectiveness for long-sequence modeling tasks.

Figure 12: Training dynamics and performance under the corrected setup. Top-left: Training loss per batch (first 3 epochs). Top-right: Gradient norm evolution (first epoch, log-scale). Bottom-left: Validation loss vs. epoch. Bottom-right: Perplexity vs. epoch. PDE-1 Step exhibits stable gradients and improved validation/perplexity versus the standard Transformer.

BibTeX


@inproceedings{zhang-zhou-2025-continuous,
  title = {Continuous-Time Attention: {PDE}-Guided Mechanisms for Long-Sequence Transformers},
  author = {Zhang, Yukun and
    Zhou, Xueqing},
  editor = {Christodoulopoulos, Christos and
    Chakraborty, Tanmoy and
    Rose, Carolyn and
    Peng, Violet},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  month = nov,
  year = {2025},
  address = {Suzhou, China},
  publisher = {Association for Computational Linguistics},
  url = {https://aclanthology.org/2025.emnlp-main.1097/},
  doi = {10.18653/v1/2025.emnlp-main.1097},
  pages = {21654--21674},
  ISBN = {979-8-89176-332-6},
  abstract = {We present Continuous-Time Attention, a novel framework that infuses partial differential equations (PDEs) into the Transformer's attention mechanism to better handle long sequences. Instead of relying on a static attention matrix, we allow attention weights to evolve along a pseudo-time dimension governed by diffusion, wave, or reaction-diffusion dynamics. This dynamic process systematically smooths local noise, strengthens long-range dependencies, and improves gradient stability during training. Our theoretical analysis shows that PDE-driven attention mitigates the exponential decay of distant interactions and improves the optimization landscape. Empirically, Continuous-Time Attention achieves consistent performance gains over both standard and long-sequence Transformer variants across a range of tasks. These results suggest that embedding continuous-time dynamics into attention mechanisms is a promising direction for enhancing global coherence and scalability in Transformer models. Code is publicly available at: https://github.com/XueqingZhou/Continuous-Time-Attention}
}