# MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models

Source: [MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models](<https://www.sota2.com/research/paper/68fa5f741e209131ce9ea018>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-21T05:00:10.757Z

Authors: Guillaume Balezo, Roger Trullo, Albert Pla Planas, Etienne Decenciere, Thomas Walter

Published: 2025-05-15T13:42:48.000Z

## Abstract

Histopathological analysis is a cornerstone of cancer diagnosis, with Hematoxylin and Eosin (H&E) staining routinely acquired for every patient to visualize cell morphology and tissue architecture. On the other hand, multiplex immunofluorescence (mIF) enables more precise cell type identification via proteomic markers, but has yet to achieve widespread clinical adoption due to cost and logistical constraints. To bridge this gap, we introduce MIPHEI (Multiplex Immunofluorescence Prediction from H&E Images), a U-Net-inspired architecture that leverages a ViT pathology foundation model as encoder to predict mIF signals from H&E images using rich pretrained representations. MIPHEI targets a comprehensive panel of markers spanning nuclear content, immune lineages (T cells, B cells, myeloid), epithelium, stroma, vasculature, and proliferation. We train our model using the publicly available OrionCRC dataset of restained H&E and mIF images from colorectal cancer tissue, and validate it on five independent datasets: HEMIT, PathoCell, IMMUcan, Lizard and PanNuke. On OrionCRC test set, MIPHEI achieves accurate cell-type classification from H&E alone, with F1 scores of 0.93 for Pan-CK, 0.83 for alpha-SMA, 0.68 for CD3e, 0.36 for CD20, and 0.28 for CD68, substantially outperforming both a state-of-the-art baseline and a random classifier for most markers. Our results indicate that, for some molecular markers, our model captures the complex relationships between nuclear morphologies in their tissue context, as visible in H&E images and molecular markers defining specific cell types. MIPHEI offers a promising step toward enabling cell-type-aware analysis of large-scale H&E datasets, in view of uncovering relationships between spatial cellular organization and patient outcomes.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multiplex Immunofluorescence Prediction on HEMIT (val & test)](<https://www.sota2.com/research/sota/multiplex-immunofluorescence-prediction-on-hemit-val-and-test.md>): Multiplex Immunofluorescence Prediction; HEMIT (val & test); F1 Score: 70.1.

- [Multiplex Immunofluorescence Prediction on ORION (test)](<https://www.sota2.com/research/sota/multiplex-immunofluorescence-prediction-on-orion-test.md>): Multiplex Immunofluorescence Prediction; ORION (test); F1: 43.8.

- [Multiplex Immunofluorescence Prediction on IMMUcan Dataset](<https://www.sota2.com/research/sota/multiplex-immunofluorescence-prediction-on-immucan-dataset.md>): Multiplex Immunofluorescence Prediction; IMMUcan Dataset; Pearson Correlation: 0.69.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2505.10294>)

[PDF](<https://arxiv.org/pdf/2505.10294.pdf>)
