News

• July, 2024: Code and model available (see Code).
• July, 2024: Web launched.

Abstract

Saliency prediction in 360º video plays an important role in modeling visual attention, and can be leveraged for content creation, compression techniques, or quality assessment methods, among others. Visual attention in immersive environments depends not only on visual input, but also on inputs from other sensory modalities, primarily audio. Despite this, only a minority of saliency prediction models have incorporated auditory inputs, and much remains to be explored about what auditory information is relevant and how to integrate it in the prediction. In this work, we propose an audiovisual saliency model for 360º video content, AViSal360. Our model integrates both spatialized and semantic audio information, together with visual inputs. We perform exhaustive comparisons to demonstrate both the actual relevance of auditory information in saliency prediction, and the superior performance of our model when compared to previous approaches.

Downloads

• Paper (SOON)

Results

The qualitative comparisons between our model, AViSal360, and the three main state-of-the-art approaches can be found in our web-based browser for the D-SAV360 dataset.

Code

You can find the code and model for AViSal360 in our GitHub repository.

Bibtex (Coming soon)

Related Work

• 2023: D-SAV360: A Dataset of Gaze Scanpaths on 360° Ambisonic Videos
@article{bernal2023d, title={D-SAV360: A Dataset of Gaze Scanpaths on 360° Ambisonic Videos}, author={Bernal-Berdun, Edurne and Martin, Daniel and Malpica, Sandra and Perez, Pedro J and Gutierrez, Diego and Masia, Belen and Serrano, Ana}, journal={IEEE Transactions on Visualization and Computer Graphics}, year={2023}, publisher={IEEE} }