News

• June, 2024: This work has obtained the CEIG 2024 Best Computer and Graphics Paper Award!
• June, 2024: Paper accepted to Computers and Graphics (Proc. CEIG 2024) (see Downloads).
• June, 2024: Demo code available (see Code).
• June, 2024: Web launched.

Abstract

Predicting the path followed by the viewer’s eyes when observing an image (a scanpath) is a challenging problem, particularly due to the inter- and intra-observer variability and the spatio-temporal dependencies of the visual attention process. Most existing approaches have focused on progressively optimizing the prediction of a gaze point given the previous ones. In this work we propose instead a probabilistic approach, which we call tSPM-Net. We build our method to account for observers’ variability by resorting to Bayesian deep learning and a probabilistic approach. Besides, we optimize our model to jointly consider both spatial and temporal dimensions of scanpaths using a novel spatio-temporal loss function based on a combination of Kullback–Leibler divergence and dynamic time warping. Our tSPM-Net yields results that outperform those of current state-of-the-art approaches, and are closer to the human baseline, suggesting that our model is able to generate scanpaths whose behavior closely resembles those of the real ones.

Downloads

• Paper

Code

You can find the code and model for ScanGAN360 in our GitHub repository.

Bibtex

Related Work

• 2020: Panoramic convolutions for 360º single-image saliency prediction
@inproceedings{martin20saliency, author={Martin, Daniel and Serrano, Ana and Masia, Belen}, title={Panoramic convolutions for $360^{\circ}$ single-image saliency prediction}, booktitle={CVPR Workshop on Computer Vision for Augmented and Virtual Reality}, year={2020} }
• 2022: ScanGAN360: A Generative Model of Realistic Scanpaths for 360 Images
@article{martin2022scangan360, title={ScanGAN360: A Generative Model of Realistic Scanpaths for 360 Images}, author={Martin, Daniel and Serrano, Ana and Bergman, Alexander W and Wetzstein, Gordon and Masia, Belen}, journal={IEEE Transactions on Visualization \& Computer Graphics}, number={01}, pages={1--1}, year={2022}, publisher={IEEE Computer Society} }
• 2020: D-SAV360: A Dataset of Gaze Scanpaths on 360° Ambisonic Videos
@article{bernal2023d, title={D-SAV360: A Dataset of Gaze Scanpaths on 360° Ambisonic Videos}, author={Bernal-Berdun, Edurne and Martin, Daniel and Malpica, Sandra and Perez, Pedro J and Gutierrez, Diego and Masia, Belen and Serrano, Ana}, journal={IEEE Transactions on Visualization and Computer Graphics}, year={2023}, publisher={IEEE} }