# MDS-ViTNet: Improving saliency prediction for Eye-Tracking with Vision Transformer

Source: [MDS-ViTNet: Improving saliency prediction for Eye-Tracking with Vision Transformer](<https://www.sota2.com/research/paper/68fa736d1e209131cea5c6f4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:00:10.820Z

Authors: Polezhaev Ignat, Goncharenko Igor, Iurina Natalya

Published: 2024-05-29T20:28:04.000Z

## Abstract

In this paper, we present a novel methodology we call MDS-ViTNet (Multi Decoder Saliency by Vision Transformer Network) for enhancing visual saliency prediction or eye-tracking. This approach holds significant potential for diverse fields, including marketing, medicine, robotics, and retail. We propose a network architecture that leverages the Vision Transformer, moving beyond the conventional ImageNet backbone. The framework adopts an encoder-decoder structure, with the encoder utilizing a Swin transformer to efficiently embed most important features. This process involves a Transfer Learning method, wherein layers from the Vision Transformer are converted by the Encoder Transformer and seamlessly integrated into a CNN Decoder. This methodology ensures minimal information loss from the original input image. The decoder employs a multi-decoding technique, utilizing dual decoders to generate two distinct attention maps. These maps are subsequently combined into a singular output via an additional CNN model. Our trained model MDS-ViTNet achieves state-of-the-art results across several benchmarks. Committed to fostering further collaboration, we intend to make our code, models, and datasets accessible to the public.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Saliency Prediction on SALICON 2017 (val)](<https://www.sota2.com/research/sota/saliency-prediction-on-salicon-2017-val.md>): Saliency Prediction; SALICON 2017 (val); AUC: 0.8684.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2405.19501>)

[PDF](<https://arxiv.org/pdf/2405.19501.pdf>)

[Implementation](<https://github.com/IgnatPolezhaev/MDS-ViTNet>)
