LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models
arXiv · HuggingFace · ▲28
Abstract (verbatim)
Recovering high-quality video from sparse event streams is a challenging task. Regression methods often blur textures, while existing generative models struggle with long-term stability. We propose LongE2V, a novel approach that leverages pre-trained video diffusion priors to jointly handle event-based video reconstruction, prediction, and frame interpolation. By fine-tuning a foundational video model, our approach achieves high data efficiency and superior perceptual quality. We introduce Autoregressive Unrolling and Adaptive Context Switching to mitigate temporal drift in extremely long sequences. We also propose Reencoding Alignment with Cross Residual Correction to ensure precise bidirectional consistency during frame interpolation. Furthermore, Event Voxel Density Augmentation ensures robustness across varying sensor resolutions. Extensive experiments on real-world benchmarks demonstrate that LongE2V outperforms state-of-the-art methods across all three tasks, exhibiting exceptional temporal coherence and zero-shot generalization. Project page: https://cdfan0627.github.io/LongE2V-page/
Background
Event cameras, inspired by biological vision, capture asynchronous brightness changes with microsecond precision and high dynamic range, making them critical for high-speed scenarios like industrial inspection or autonomous driving. The core challenge is reconstructing high-fidelity videos from sparse, intensity-free event streams—a task complicated by real-world demands for detail preservation, long-term coherence, and zero-shot frame interpolation.
Previous approaches faced fundamental limitations: regression-based models (e.g., E2VID) suffered from "regression-to-the-mean," blurring textures, while naive diffusion models accumulated errors in long sequences and produced ghosting artifacts in interpolation. Early methods relied on task-specific architectures (CNNs/RNNs or Transformers), forcing trade-offs between reconstruction, prediction, and interpolation. These failures stemmed from event data’s sparsity and asynchronicity, which traditional algorithms struggled to process consistently over time.
LongE2V addresses these issues by leveraging pre-trained video diffusion models (VDMs) as a unified framework. Its key innovations include: 1) Treating reconstruction, prediction, and interpolation as conditional generation tasks driven by event voxels; 2) Introducing "Autoregressive Unrolling" and "Adaptive Context Switching" to dynamically adjust temporal dependencies, mitigating drift in long sequences; 3) Using "Reencoding Alignment with Cross Residual Correction" to ensure temporal consistency in interpolation. A novel "Event Voxel Density Augmentation" further enhances robustness across sensor resolutions.
Unlike prior work, LongE2V uniquely unifies multiple tasks under a single diffusion-based paradigm, avoids task-specific architectures, and explicitly tackles long-term stability through dynamic temporal modeling. Experiments show it outperforms state-of-the-art methods in all three tasks, achieving superior perceptual quality, stability, and zero-shot generalization—addressing gaps left by both traditional methods and naive generative approaches.




